[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:55 dist-test-slave-25dq krb5kdc[817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:55 dist-test-slave-25dq krb5kdc[817](info): set up 2 sockets
Sep 27 21:38:55 dist-test-slave-25dq krb5kdc[817](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 21:38:56.997954 810 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:38:57 dist-test-slave-25dq krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:38:57.181210 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40141
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:57.525075 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:57.525485 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:57.525640 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:57.533320 830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:57.533470 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:57.533586 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:57.533694 830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:57.533769 830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:57.544090 830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:57.545497 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:57.546907 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:57.559545 836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:57.559584 838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:57.559504 835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:57.560745 830 server_base.cc:1034] running on GCE node
I20260927 21:38:57.561688 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:57.563318 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:57.564590 830 hybrid_clock.cc:648] HybridClock initialized: now 1790545137564469 us; error 105 us; skew 500 ppm
I20260927 21:38:57.564988 830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:57 dist-test-slave-25dq krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:57.568574 830 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:38:57.570760 830 webserver.cc:460] Webserver started at http://127.0.202.190:34889/ using document root <none> and password file <none>
I20260927 21:38:57.571532 830 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:57.571667 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:57.572016 830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:57.575414 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "0810c9ccbdc6400f913ee47de5913031"
format_stamp: "Formatted at 2026-09-27 21:38:57 on dist-test-slave-25dq"
server_key: "d71e30a9aeb28605875029a37b9bf1e3"
server_key_iv: "a498dce03c8b85afe6617a5b0c23ec74"
server_key_version: "encryptionkey@0"
I20260927 21:38:57.576268 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "0810c9ccbdc6400f913ee47de5913031"
format_stamp: "Formatted at 2026-09-27 21:38:57 on dist-test-slave-25dq"
server_key: "d71e30a9aeb28605875029a37b9bf1e3"
server_key_iv: "a498dce03c8b85afe6617a5b0c23ec74"
server_key_version: "encryptionkey@0"
I20260927 21:38:57.582108 830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:38:57.585781 845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:57.587142 830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 21:38:57.587332 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "0810c9ccbdc6400f913ee47de5913031"
format_stamp: "Formatted at 2026-09-27 21:38:57 on dist-test-slave-25dq"
server_key: "d71e30a9aeb28605875029a37b9bf1e3"
server_key_iv: "a498dce03c8b85afe6617a5b0c23ec74"
server_key_version: "encryptionkey@0"
I20260927 21:38:57.587538 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:57.600955 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:57.605499 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:57.605803 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:57.626830 830 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40141
I20260927 21:38:57.626849 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40141 every 8 connection(s)
I20260927 21:38:57.628341 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:57.629788 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 830
I20260927 21:38:57.630119 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:38:57.632674 810 external_mini_cluster.cc:1428] Setting key fd341a838498ac2fad7a038951b1dbc9
I20260927 21:38:57.633898 897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:57.644959 897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031: Bootstrap starting.
I20260927 21:38:57.648509 897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:57.649893 897 log.cc:824] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:57.653299 897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031: No bootstrap required, opened a new log
I20260927 21:38:57.664515 897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0810c9ccbdc6400f913ee47de5913031" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40141 } }
I20260927 21:38:57.665027 897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:57.665134 897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0810c9ccbdc6400f913ee47de5913031, State: Initialized, Role: FOLLOWER
Sep 27 21:38:57 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:38:57.665959 897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0810c9ccbdc6400f913ee47de5913031" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40141 } }
I20260927 21:38:57.666266 897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:57.666414 897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:57.666600 897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:57.669577 897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0810c9ccbdc6400f913ee47de5913031" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40141 } }
I20260927 21:38:57.670148 897 leader_election.cc:302] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0810c9ccbdc6400f913ee47de5913031; no voters:
I20260927 21:38:57.670836 897 leader_election.cc:288] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:57.670992 902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:57.673169 897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:57.677976 902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [term 1 LEADER]: Becoming Leader. State: Replica: 0810c9ccbdc6400f913ee47de5913031, State: Running, Role: LEADER
I20260927 21:38:57.678546 902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0810c9ccbdc6400f913ee47de5913031" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40141 } }
I20260927 21:38:57.678939 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:57.637481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.0.202.190:40141)
0927 21:38:57.638395 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:57.638411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:57.639709 (+ 1298us) server_negotiation.cc:407] Connection header received
0927 21:38:57.649403 (+ 9694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:57.649432 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:57.649953 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:57.650412 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:57.654227 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:57.656317 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:57.658823 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:57.659716 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:57.659884 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:57.667520 (+ 7636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:57.667528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:57.667547 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:57.667597 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:57.671849 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:57.673090 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:57.673104 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:57.673115 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:57.673393 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:57.674015 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:57.674023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:57.674027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:57.674813 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:57.675180 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:57.676513 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0927 21:38:57.677056 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":399,"threads_started":1}
I20260927 21:38:57.682210 904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0810c9ccbdc6400f913ee47de5913031. Latest consensus state: current_term: 1 leader_uuid: "0810c9ccbdc6400f913ee47de5913031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0810c9ccbdc6400f913ee47de5913031" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40141 } } }
I20260927 21:38:57.683934 904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:57.681996 903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0810c9ccbdc6400f913ee47de5913031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0810c9ccbdc6400f913ee47de5913031" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40141 } } }
I20260927 21:38:57.684628 903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:57.690207 911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:57.696460 911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:57.707073 911 catalog_manager.cc:1347] Generated new cluster ID: d66cfc03131b42c4b3ab1b316525b333
I20260927 21:38:57.707202 911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:57.725081 911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:57.727139 911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:57.733955 911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0810c9ccbdc6400f913ee47de5913031: Generated new TSK 0
I20260927 21:38:57.734817 911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:38:57.792083 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40141
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:58.146948 923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:58.147238 923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:58.147415 923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:58.154881 923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:58.154990 923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:58.155200 923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:38:58.165706 923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:58.167096 923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:58.168921 923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:58.185583 929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:58.185667 931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:58.185601 928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:58.186450 923 server_base.cc:1034] running on GCE node
I20260927 21:38:58.187067 923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:58.188056 923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:58.189249 923 hybrid_clock.cc:648] HybridClock initialized: now 1790545138189256 us; error 109 us; skew 500 ppm
I20260927 21:38:58.189620 923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:58 dist-test-slave-25dq krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:58.193064 923 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:38:58.194653 923 webserver.cc:460] Webserver started at http://127.0.202.129:38411/ using document root <none> and password file <none>
I20260927 21:38:58.195441 923 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:58.195657 923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:58.196081 923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:58.199942 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "10c612d61dc04542bb70d896f9c770da"
format_stamp: "Formatted at 2026-09-27 21:38:58 on dist-test-slave-25dq"
server_key: "03b1cb9409eae270dc92a99b4fc602e1"
server_key_iv: "335fb2a6fb0eefbf87a827fe482bb745"
server_key_version: "encryptionkey@0"
I20260927 21:38:58.200841 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "10c612d61dc04542bb70d896f9c770da"
format_stamp: "Formatted at 2026-09-27 21:38:58 on dist-test-slave-25dq"
server_key: "03b1cb9409eae270dc92a99b4fc602e1"
server_key_iv: "335fb2a6fb0eefbf87a827fe482bb745"
server_key_version: "encryptionkey@0"
I20260927 21:38:58.206391 923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:38:58.210599 938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:58.212399 923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 21:38:58.212596 923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "10c612d61dc04542bb70d896f9c770da"
format_stamp: "Formatted at 2026-09-27 21:38:58 on dist-test-slave-25dq"
server_key: "03b1cb9409eae270dc92a99b4fc602e1"
server_key_iv: "335fb2a6fb0eefbf87a827fe482bb745"
server_key_version: "encryptionkey@0"
I20260927 21:38:58.212839 923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:58.233568 923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:58.238971 923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:58.239245 923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:58.240236 923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:38:58.241895 923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:38:58.241991 923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:58.242129 923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:38:58.242197 923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:58.282924 923 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:43113
I20260927 21:38:58.282934 1050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:43113 every 8 connection(s)
I20260927 21:38:58.284219 923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:38:58.291865 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 923
I20260927 21:38:58.292114 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:38:58.292893 810 external_mini_cluster.cc:1428] Setting key 299be1be23c0c85af6b883b165ec28cb
Sep 27 21:38:58 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:38:58.308182 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:58.287138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:44021 (local address 127.0.202.190:40141)
0927 21:38:58.287637 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:58.287644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:58.288757 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 21:38:58.290700 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:58.290706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:58.290821 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:58.290943 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:58.293097 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.294128 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.296525 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.296927 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.297034 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:58.301868 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:58.301873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:58.301876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:58.301910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:58.304610 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:58.305621 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:58.305626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:58.305629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:58.305765 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:58.306342 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:58.306346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:58.306348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:58.306631 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:58.306745 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:58.307724 (+ 979us) server_negotiation.cc:299] Negotiation successful
0927 21:38:58.307935 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":216,"threads_started":1}
I20260927 21:38:58.309914 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:58.287789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40141 (local address 127.0.202.129:44021)
0927 21:38:58.288595 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:58.288634 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:58.290394 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:58.291161 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:58.291178 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:58.291976 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:58.292869 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:58.292898 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.294415 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.294421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:58.296347 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:58.296365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.297180 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.297190 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:58.297544 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:58.298704 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:58.298743 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:58.301595 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:58.304750 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:58.304763 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:58.304786 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:58.305432 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:58.305905 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:58.305912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:58.305919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:58.306174 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:58.306772 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:58.306783 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:58.307293 (+ 510us) client_negotiation.cc:769] Sending connection context
0927 21:38:58.307624 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 21:38:58.308285 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":286,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:38:58.311791 1051 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40141
I20260927 21:38:58.312101 1051 heartbeater.cc:464] Registering TS with master...
I20260927 21:38:58.312929 1051 heartbeater.cc:511] Master 127.0.202.190:40141 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:38:58.315356 862 ts_manager.cc:194] Registered new tserver with Master: 10c612d61dc04542bb70d896f9c770da (127.0.202.129:43113)
I20260927 21:38:58.316939 862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:44021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:38:58.337450 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40141
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:58.687170 1057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:58.687530 1057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:58.687721 1057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:58.695968 1057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:58.696137 1057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:58.696381 1057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:38:58.706652 1057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:58.708012 1057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:58.709477 1057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:58.724647 1063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:58.725342 1065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:58.725401 1062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:58.725899 1057 server_base.cc:1034] running on GCE node
I20260927 21:38:58.726550 1057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:58.727635 1057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:58.728873 1057 hybrid_clock.cc:648] HybridClock initialized: now 1790545138728812 us; error 53 us; skew 500 ppm
I20260927 21:38:58.729204 1057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:58 dist-test-slave-25dq krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:58.732616 1057 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:38:58.734650 1057 webserver.cc:460] Webserver started at http://127.0.202.130:45613/ using document root <none> and password file <none>
I20260927 21:38:58.735337 1057 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:58.735441 1057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:58.735700 1057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:58.739288 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "112c970ff20741cc81fcb59440c73fa0"
format_stamp: "Formatted at 2026-09-27 21:38:58 on dist-test-slave-25dq"
server_key: "7c9291a34edf1e0cb2671846cf1cba56"
server_key_iv: "812fdb1b1f7613a877f8ef6eba470466"
server_key_version: "encryptionkey@0"
I20260927 21:38:58.740063 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "112c970ff20741cc81fcb59440c73fa0"
format_stamp: "Formatted at 2026-09-27 21:38:58 on dist-test-slave-25dq"
server_key: "7c9291a34edf1e0cb2671846cf1cba56"
server_key_iv: "812fdb1b1f7613a877f8ef6eba470466"
server_key_version: "encryptionkey@0"
I20260927 21:38:58.745636 1057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:38:58.749310 1072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:58.750788 1057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:38:58.750932 1057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "112c970ff20741cc81fcb59440c73fa0"
format_stamp: "Formatted at 2026-09-27 21:38:58 on dist-test-slave-25dq"
server_key: "7c9291a34edf1e0cb2671846cf1cba56"
server_key_iv: "812fdb1b1f7613a877f8ef6eba470466"
server_key_version: "encryptionkey@0"
I20260927 21:38:58.751112 1057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:58.770042 1057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:58.775290 1057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:58.775609 1057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:58.776753 1057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:38:58.778216 1057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:38:58.778303 1057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:58.778465 1057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:38:58.778532 1057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:58.818279 1057 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:45871
I20260927 21:38:58.818367 1184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:45871 every 8 connection(s)
I20260927 21:38:58.819644 1057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:38:58.827092 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 1057
I20260927 21:38:58.827317 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:38:58.827834 810 external_mini_cluster.cc:1428] Setting key 56b8bb8964f53426984d326ce536907c
Sep 27 21:38:58 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:38:58.845842 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:58.822466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:51181 (local address 127.0.202.190:40141)
0927 21:38:58.822932 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:58.822939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:58.824059 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 21:38:58.826343 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:58.826348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:58.826421 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:58.826530 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:58.828672 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.829754 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.832893 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.833570 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.833687 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:58.837621 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:58.837631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:58.837641 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:58.837692 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:58.841802 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:58.842777 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:58.842786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:58.842793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:58.842984 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:58.843536 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:58.843544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:58.843551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:58.843997 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:58.844185 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:58.845112 (+ 927us) server_negotiation.cc:299] Negotiation successful
0927 21:38:58.845384 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":216,"threads_started":1}
I20260927 21:38:58.846729 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:58.823135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40141 (local address 127.0.202.130:51181)
0927 21:38:58.823883 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:58.823927 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:58.826050 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:58.826833 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:58.826846 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:58.827610 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:58.828477 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:58.828506 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.829964 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.829971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:58.832350 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:58.832369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:58.833827 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:58.833836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:58.834177 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:58.834993 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:58.835029 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:58.837336 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:58.841978 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:58.841985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:58.842009 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:58.842605 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:58.843131 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:58.843134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:58.843203 (+ 69us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:58.843398 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:58.844162 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:58.844169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:58.844658 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 21:38:58.844945 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 21:38:58.845462 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":291,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:38:58.848590 1185 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40141
I20260927 21:38:58.848961 1185 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:38:58.849988 1185 heartbeater.cc:511] Master 127.0.202.190:40141 requested a full tablet report, sending...
I20260927 21:38:58.851889 862 ts_manager.cc:194] Registered new tserver with Master: 112c970ff20741cc81fcb59440c73fa0 (127.0.202.130:45871)
I20260927 21:38:58.853312 862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:51181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:38:58.871807 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40141
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:59.214102 1191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:59.214396 1191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:59.214491 1191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:59.222605 1191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:59.222735 1191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:59.222851 1191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:38:59.232745 1191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:59.233999 1191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:59.235294 1191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:59.249007 1196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:59.249025 1197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:59.249050 1199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:59.250291 1191 server_base.cc:1034] running on GCE node
I20260927 21:38:59.250931 1191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:59.251917 1191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:59.253157 1191 hybrid_clock.cc:648] HybridClock initialized: now 1790545139253116 us; error 46 us; skew 500 ppm
I20260927 21:38:59.253556 1191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:59.257798 1191 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:38:59.259459 1191 webserver.cc:460] Webserver started at http://127.0.202.131:34793/ using document root <none> and password file <none>
I20260927 21:38:59.260192 1191 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:59.260353 1191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:59.260684 1191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:59.263785 1191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "e5728725c8bb49b69f6e8266b9f129f7"
format_stamp: "Formatted at 2026-09-27 21:38:59 on dist-test-slave-25dq"
server_key: "557ee2a042a6df01efe9d4a7b5a317b9"
server_key_iv: "28598c262679e65d29996b8ffcb489b5"
server_key_version: "encryptionkey@0"
I20260927 21:38:59.264640 1191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "e5728725c8bb49b69f6e8266b9f129f7"
format_stamp: "Formatted at 2026-09-27 21:38:59 on dist-test-slave-25dq"
server_key: "557ee2a042a6df01efe9d4a7b5a317b9"
server_key_iv: "28598c262679e65d29996b8ffcb489b5"
server_key_version: "encryptionkey@0"
I20260927 21:38:59.269994 1191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:38:59.273968 1206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:59.275660 1191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:38:59.275862 1191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "e5728725c8bb49b69f6e8266b9f129f7"
format_stamp: "Formatted at 2026-09-27 21:38:59 on dist-test-slave-25dq"
server_key: "557ee2a042a6df01efe9d4a7b5a317b9"
server_key_iv: "28598c262679e65d29996b8ffcb489b5"
server_key_version: "encryptionkey@0"
I20260927 21:38:59.276113 1191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:59.307435 1191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:59.312443 1191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:59.312788 1191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:59.313876 1191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:38:59.315286 1191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:38:59.315361 1191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:59.315498 1191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:38:59.315548 1191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:59.320026 1051 heartbeater.cc:503] Master 127.0.202.190:40141 was elected leader, sending a full tablet report...
I20260927 21:38:59.353670 1191 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:44363
I20260927 21:38:59.353714 1318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:44363 every 8 connection(s)
I20260927 21:38:59.355010 1191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:38:59.360944 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 1191
I20260927 21:38:59.361141 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:38:59.361593 810 external_mini_cluster.cc:1428] Setting key 7f54c88a688cf52bc5c3fe8d9f893d93
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:38:59.379240 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:59.358065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:53681 (local address 127.0.202.190:40141)
0927 21:38:59.358546 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:59.358551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:59.359408 (+ 857us) server_negotiation.cc:407] Connection header received
0927 21:38:59.361533 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:59.361539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:59.361681 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:59.361810 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:59.363865 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.365151 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.366891 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.367296 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.367439 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:59.371169 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:59.371175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:59.371182 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:59.371218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:59.375399 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.376222 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.376230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.376237 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.376519 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.377118 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.377123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.377127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.377513 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:59.377638 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:59.378613 (+ 975us) server_negotiation.cc:299] Negotiation successful
0927 21:38:59.378844 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":252,"threads_started":1}
I20260927 21:38:59.380477 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:59.358541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40141 (local address 127.0.202.131:53681)
0927 21:38:59.359240 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:59.359287 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:59.361238 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:59.362060 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:59.362068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:59.362867 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:59.363688 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:59.363708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.365309 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.365314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:59.366706 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:59.366719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.367553 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.367561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:59.367809 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:59.368623 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:59.368651 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:59.370928 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:59.375597 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:59.375605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:59.375622 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:59.376088 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:59.376664 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:59.376670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:59.376674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:59.376997 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:59.377665 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:59.377674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:59.378143 (+ 469us) client_negotiation.cc:769] Sending connection context
0927 21:38:59.378539 (+ 396us) client_negotiation.cc:240] Negotiation successful
0927 21:38:59.379094 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":267,"threads_started":1}
I20260927 21:38:59.381934 1319 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40141
I20260927 21:38:59.382292 1319 heartbeater.cc:464] Registering TS with master...
I20260927 21:38:59.383056 1319 heartbeater.cc:511] Master 127.0.202.190:40141 requested a full tablet report, sending...
I20260927 21:38:59.384617 862 ts_manager.cc:194] Registered new tserver with Master: e5728725c8bb49b69f6e8266b9f129f7 (127.0.202.131:44363)
I20260927 21:38:59.385385 862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:53681
I20260927 21:38:59.391290 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:38:59.438211 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:59.421651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.0.202.129:43113)
0927 21:38:59.422379 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:59.422387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:59.422418 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:38:59.422497 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:59.422504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:59.422722 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:59.422851 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:59.424254 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.425486 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.426854 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.427624 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.427765 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:59.432096 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:59.432109 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:59.432125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:59.432170 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:59.435253 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.435928 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.435936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.435939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.436065 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.436610 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.436614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.436616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.436969 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:59.437199 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:59.437559 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 21:38:59.437802 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":365,"threads_started":1}
W20260927 21:38:59.439643 966 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51192
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:38:59.485829 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:59.470561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.0.202.129:43113)
0927 21:38:59.470892 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:59.470899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:59.471169 (+ 270us) server_negotiation.cc:407] Connection header received
0927 21:38:59.471392 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:59.471397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:59.471490 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:59.471602 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:59.472986 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.473683 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.474943 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.475354 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.475488 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:59.479660 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:59.479665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:59.479668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:59.479703 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:59.482965 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.483731 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.483737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.483741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.483883 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.484284 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.484331 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.484338 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.484632 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:59.484895 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:59.485238 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 21:38:59.485484 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260927 21:38:59.487877 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 923
I20260927 21:38:59.501789 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:59.489088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.0.202.129:43113)
0927 21:38:59.489439 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:59.489446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:59.489619 (+ 173us) server_negotiation.cc:407] Connection header received
0927 21:38:59.489854 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:59.489860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:59.489925 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:59.490026 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:59.491384 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.492110 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.493420 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.493836 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.493969 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:59.496224 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:59.496229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:59.496232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:59.496264 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:59.498942 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.499606 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.499612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.499616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.499756 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.500188 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.500194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.500198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.500553 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:59.500801 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:59.501160 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 21:38:59.501383 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 21:38:59.502696 1046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:59.587350 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 923
I20260927 21:38:59.604482 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 1057
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:38:59.621495 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:59.605566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.0.202.130:45871)
0927 21:38:59.606154 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:59.606161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:59.606188 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:38:59.606301 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:59.606307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:59.606521 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:59.606651 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:59.607848 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.608935 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.610264 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.610839 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.610942 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:59.615320 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:59.615336 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:59.615351 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:59.615398 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:59.618461 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.619164 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.619173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.619175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.619305 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.619710 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.619714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.619716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.620068 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:59.620466 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:59.620849 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 21:38:59.621066 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":322,"threads_started":1}
I20260927 21:38:59.622911 1180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:59.713651 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 1057
I20260927 21:38:59.736091 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 1191
Sep 27 21:38:59 dist-test-slave-25dq krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:38:59.753492 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:59.737311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.0.202.131:44363)
0927 21:38:59.737959 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:59.737967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:59.737997 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:38:59.738058 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:59.738064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:59.738285 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:59.738416 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:59.739718 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.741129 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.742523 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:59.743153 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:59.743261 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:59.747596 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:59.747608 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:59.747623 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:59.747659 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:59.750609 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.751353 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.751369 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.751372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.751490 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:59.751857 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:59.751863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:59.751866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:59.752208 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:59.752545 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:59.752883 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 21:38:59.753149 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":317,"threads_started":1}
I20260927 21:38:59.754827 1314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:59.838510 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 1191
I20260927 21:38:59.854889 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 830
I20260927 21:38:59.856402 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:59.963726 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 830
2026-09-27T21:38:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4124 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:00 dist-test-slave-25dq krb5kdc[1341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:00 dist-test-slave-25dq krb5kdc[1341](info): set up 2 sockets
Sep 27 21:39:00 dist-test-slave-25dq krb5kdc[1341](info): commencing operation
krb5kdc: starting...
W20260927 21:39:01.116756 810 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:01 dist-test-slave-25dq krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:01.313112 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41359
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:41359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:01.641178 1354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:01.641497 1354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:01.641584 1354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:01.649209 1354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:01.649680 1354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:01.649788 1354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:01.650018 1354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:01.650220 1354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:01.661357 1354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:41359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:01.662686 1354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:01.663951 1354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:01.676105 1362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:01.676188 1359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:01.676188 1360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:01.677719 1354 server_base.cc:1034] running on GCE node
I20260927 21:39:01.678579 1354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:01.680060 1354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:01.681288 1354 hybrid_clock.cc:648] HybridClock initialized: now 1790545141681203 us; error 76 us; skew 500 ppm
I20260927 21:39:01.681638 1354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:01 dist-test-slave-25dq krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:01.690680 1354 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:01.692554 1354 webserver.cc:460] Webserver started at http://127.0.202.190:37693/ using document root <none> and password file <none>
I20260927 21:39:01.693336 1354 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:01.693491 1354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:01.693825 1354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:01.696988 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "9308670f7ad84c00a6dba51a8a2dcf7d"
format_stamp: "Formatted at 2026-09-27 21:39:01 on dist-test-slave-25dq"
server_key: "1c48b2bd184706fe695fd6ea5f0761d5"
server_key_iv: "1b0d2fcbb0de01f8d714c2b028e637e0"
server_key_version: "encryptionkey@0"
I20260927 21:39:01.697804 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "9308670f7ad84c00a6dba51a8a2dcf7d"
format_stamp: "Formatted at 2026-09-27 21:39:01 on dist-test-slave-25dq"
server_key: "1c48b2bd184706fe695fd6ea5f0761d5"
server_key_iv: "1b0d2fcbb0de01f8d714c2b028e637e0"
server_key_version: "encryptionkey@0"
I20260927 21:39:01.703310 1354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:39:01.707499 1369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:01.709059 1354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:01.709273 1354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "9308670f7ad84c00a6dba51a8a2dcf7d"
format_stamp: "Formatted at 2026-09-27 21:39:01 on dist-test-slave-25dq"
server_key: "1c48b2bd184706fe695fd6ea5f0761d5"
server_key_iv: "1b0d2fcbb0de01f8d714c2b028e637e0"
server_key_version: "encryptionkey@0"
I20260927 21:39:01.709580 1354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:01.744567 1354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:01.749396 1354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:01.749730 1354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:01.770766 1354 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:41359
I20260927 21:39:01.770785 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:41359 every 8 connection(s)
I20260927 21:39:01.772395 1354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:01.776463 1421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:01.782645 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 1354
I20260927 21:39:01.782910 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:01.783713 810 external_mini_cluster.cc:1428] Setting key 36629897326d2cd44375fcc0752d4bff
I20260927 21:39:01.788414 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d: Bootstrap starting.
I20260927 21:39:01.791800 1421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:01.792985 1421 log.cc:824] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:01.796483 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d: No bootstrap required, opened a new log
Sep 27 21:39:01 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:01.805859 1421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41359 } }
I20260927 21:39:01.806339 1421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:01.806463 1421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9308670f7ad84c00a6dba51a8a2dcf7d, State: Initialized, Role: FOLLOWER
I20260927 21:39:01.807262 1421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41359 } }
I20260927 21:39:01.807595 1421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:01.807762 1421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:01.807986 1421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:01.810698 1421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41359 } }
I20260927 21:39:01.810829 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:01.786286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.0.202.190:41359)
0927 21:39:01.787268 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:01.787282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:01.787332 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:39:01.789938 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:01.789969 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:01.790443 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:01.790899 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:01.792758 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:01.794904 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:01.796358 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:01.797189 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:01.797443 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:01.802013 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:01.802023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:01.802041 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:01.802089 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:01.805853 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:01.806961 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:01.806970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:01.806979 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:01.807266 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:01.807735 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:01.807742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:01.807746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:01.808514 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:01.808862 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:01.809360 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 21:39:01.809797 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":494,"threads_started":1}
I20260927 21:39:01.811408 1421 leader_election.cc:302] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9308670f7ad84c00a6dba51a8a2dcf7d; no voters:
I20260927 21:39:01.812093 1421 leader_election.cc:288] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:01.812211 1426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:01.814426 1421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:01.815024 1426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [term 1 LEADER]: Becoming Leader. State: Replica: 9308670f7ad84c00a6dba51a8a2dcf7d, State: Running, Role: LEADER
I20260927 21:39:01.815531 1426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41359 } }
I20260927 21:39:01.818873 1428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9308670f7ad84c00a6dba51a8a2dcf7d. Latest consensus state: current_term: 1 leader_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41359 } } }
I20260927 21:39:01.819021 1427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9308670f7ad84c00a6dba51a8a2dcf7d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41359 } } }
I20260927 21:39:01.819389 1427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:01.819276 1428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:01.820871 1437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:01.825994 1437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:01.836150 1437 catalog_manager.cc:1347] Generated new cluster ID: 7272ccf042124b6687db4da06de9d387
I20260927 21:39:01.836320 1437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:01.857703 1437 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:01.859078 1437 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:01.866876 1437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9308670f7ad84c00a6dba51a8a2dcf7d: Generated new TSK 0
I20260927 21:39:01.867775 1437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:01.925055 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41359
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:02.273204 1447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:02.273559 1447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:02.273730 1447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:02.281190 1447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:02.281379 1447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:02.281561 1447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:02.291782 1447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:02.293398 1447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:02.294793 1447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:02.308990 1455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:02.309064 1453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:02.309087 1452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:02.309880 1447 server_base.cc:1034] running on GCE node
I20260927 21:39:02.310565 1447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:02.311740 1447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:02.312989 1447 hybrid_clock.cc:648] HybridClock initialized: now 1790545142312940 us; error 42 us; skew 500 ppm
I20260927 21:39:02.313361 1447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:02 dist-test-slave-25dq krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:02.316979 1447 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:02.318642 1447 webserver.cc:460] Webserver started at http://127.0.202.129:35845/ using document root <none> and password file <none>
I20260927 21:39:02.319327 1447 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:02.319437 1447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:02.319700 1447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:02.322883 1447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "e4d324728ce94428ad7e5f74a2b0c4e2"
format_stamp: "Formatted at 2026-09-27 21:39:02 on dist-test-slave-25dq"
server_key: "75fb85beb8e38cc5575ca1f51b8bbe86"
server_key_iv: "d6a62493cc6721fad525f6298393a2a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:02.323700 1447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "e4d324728ce94428ad7e5f74a2b0c4e2"
format_stamp: "Formatted at 2026-09-27 21:39:02 on dist-test-slave-25dq"
server_key: "75fb85beb8e38cc5575ca1f51b8bbe86"
server_key_iv: "d6a62493cc6721fad525f6298393a2a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:02.329146 1447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 21:39:02.332888 1462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:02.334605 1447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:39:02.334820 1447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "e4d324728ce94428ad7e5f74a2b0c4e2"
format_stamp: "Formatted at 2026-09-27 21:39:02 on dist-test-slave-25dq"
server_key: "75fb85beb8e38cc5575ca1f51b8bbe86"
server_key_iv: "d6a62493cc6721fad525f6298393a2a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:02.335067 1447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:02.351372 1447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:02.356556 1447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:02.356902 1447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:02.358021 1447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:02.359495 1447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:02.359575 1447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:02.359717 1447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:02.359805 1447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:02.398540 1447 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:44439
I20260927 21:39:02.398654 1574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:44439 every 8 connection(s)
I20260927 21:39:02.400087 1447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:02.405617 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 1447
I20260927 21:39:02.405881 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:02.406632 810 external_mini_cluster.cc:1428] Setting key 5fd1af9492c9a6ef7d768bdf31a194ac
Sep 27 21:39:02 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:02.422845 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:02.403027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:48993 (local address 127.0.202.190:41359)
0927 21:39:02.403505 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:02.403511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:02.404754 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 21:39:02.406961 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:02.406967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:02.407068 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:02.407211 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:02.409123 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.409961 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.411689 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.412123 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.412279 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:02.416459 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:02.416464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:02.416468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:02.416501 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:02.419455 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:02.420198 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:02.420203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:02.420207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:02.420581 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:02.421088 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:02.421093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:02.421096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:02.421365 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:02.421534 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:02.422259 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 21:39:02.422485 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:39:02.423954 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:02.403660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41359 (local address 127.0.202.129:48993)
0927 21:39:02.404593 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:02.404635 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:02.406588 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:02.407501 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:02.407511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:02.408071 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:02.408903 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:02.408925 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.410147 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.410151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:02.411520 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:02.411532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.412418 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.412427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:02.412700 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:02.413637 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:02.413661 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:02.416189 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:02.419608 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:02.419617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:02.419637 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:02.420066 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:02.420707 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:02.420713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:02.420717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:02.420938 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:02.421499 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:02.421511 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:02.421875 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 21:39:02.422150 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 21:39:02.422767 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":301,"threads_started":1}
I20260927 21:39:02.426009 1575 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:39:02.426532 1575 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:02.427501 1575 heartbeater.cc:511] Master 127.0.202.190:41359 requested a full tablet report, sending...
I20260927 21:39:02.430049 1386 ts_manager.cc:194] Registered new tserver with Master: e4d324728ce94428ad7e5f74a2b0c4e2 (127.0.202.129:44439)
I20260927 21:39:02.431651 1386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:48993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:02.453114 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41359
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:02.791422 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:02.791704 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:02.791870 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:02.799343 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:02.799471 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:02.799667 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:02.809784 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:02.811146 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:02.812572 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:02.825632 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:02.825632 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:02.825609 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:02.826340 1581 server_base.cc:1034] running on GCE node
I20260927 21:39:02.826972 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:02.827945 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:02.829209 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790545142829165 us; error 54 us; skew 500 ppm
I20260927 21:39:02.829577 1581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:02 dist-test-slave-25dq krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:02.832994 1581 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:02.834636 1581 webserver.cc:460] Webserver started at http://127.0.202.130:37219/ using document root <none> and password file <none>
I20260927 21:39:02.835353 1581 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:02.835466 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:02.835815 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:02.838908 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "52437120dc0246e0a9eb75a9d69e3d17"
format_stamp: "Formatted at 2026-09-27 21:39:02 on dist-test-slave-25dq"
server_key: "7104588a69d71a9f4a0fe6fa64249ba3"
server_key_iv: "5a77ca3b589bb9edc20fb5d387ab7b55"
server_key_version: "encryptionkey@0"
I20260927 21:39:02.839767 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "52437120dc0246e0a9eb75a9d69e3d17"
format_stamp: "Formatted at 2026-09-27 21:39:02 on dist-test-slave-25dq"
server_key: "7104588a69d71a9f4a0fe6fa64249ba3"
server_key_iv: "5a77ca3b589bb9edc20fb5d387ab7b55"
server_key_version: "encryptionkey@0"
I20260927 21:39:02.845736 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:39:02.849661 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:02.851315 1581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:02.851531 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "52437120dc0246e0a9eb75a9d69e3d17"
format_stamp: "Formatted at 2026-09-27 21:39:02 on dist-test-slave-25dq"
server_key: "7104588a69d71a9f4a0fe6fa64249ba3"
server_key_iv: "5a77ca3b589bb9edc20fb5d387ab7b55"
server_key_version: "encryptionkey@0"
I20260927 21:39:02.851749 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:02.881974 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:02.887069 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:02.887404 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:02.888690 1581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:02.890193 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:02.890311 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:02.890489 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:02.890570 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:02.929020 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:33951
I20260927 21:39:02.929068 1708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:33951 every 8 connection(s)
I20260927 21:39:02.930554 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:02.932849 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 1581
I20260927 21:39:02.933053 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:02.933643 810 external_mini_cluster.cc:1428] Setting key 5b2e72a043fd30b56025ccd04e0eb189
Sep 27 21:39:02 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:39:02.954789 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:02.933617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:33149 (local address 127.0.202.190:41359)
0927 21:39:02.936723 (+ 3106us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:02.936730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:02.936751 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:39:02.937701 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:02.937708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:02.937794 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:02.937923 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:02.939631 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.940860 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.942567 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.943018 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.943132 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:02.948195 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:02.948200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:02.948204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:02.948246 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:02.951457 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:02.952334 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:02.952344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:02.952347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:02.952477 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:02.952947 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:02.952953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:02.952956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:02.953240 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:02.953391 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:02.954182 (+ 791us) server_negotiation.cc:299] Negotiation successful
0927 21:39:02.954409 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2985,"thread_start_us":2889,"threads_started":1}
I20260927 21:39:02.956280 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:02.934406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41359 (local address 127.0.202.130:33149)
0927 21:39:02.935344 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:02.935391 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:02.937358 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:02.938204 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:02.938213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:02.938736 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:02.939443 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:02.939464 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.941048 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.941053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:02.942396 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:02.942408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:02.943278 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:02.943288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:02.943629 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:02.944643 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:02.944679 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:02.947902 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:02.951668 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:02.951678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:02.951707 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:02.952160 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:02.952607 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:02.952616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:02.952620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:02.952830 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:02.953382 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:02.953390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:02.953762 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 21:39:02.954096 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 21:39:02.954683 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":457,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:39:02.958154 1709 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41359
I20260927 21:39:02.958498 1709 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:02.959239 1709 heartbeater.cc:511] Master 127.0.202.190:41359 requested a full tablet report, sending...
I20260927 21:39:02.960933 1386 ts_manager.cc:194] Registered new tserver with Master: 52437120dc0246e0a9eb75a9d69e3d17 (127.0.202.130:33951)
I20260927 21:39:02.961793 1386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:33149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:02.984102 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41359
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:03.319866 1715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:03.320154 1715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:03.320250 1715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:03.327579 1715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:03.327705 1715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:03.327819 1715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:03.337920 1715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:03.339272 1715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:03.340777 1715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:03.355940 1721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:03.356184 1720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:03.356189 1723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:03.357333 1715 server_base.cc:1034] running on GCE node
I20260927 21:39:03.358098 1715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:03.359165 1715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:03.360496 1715 hybrid_clock.cc:648] HybridClock initialized: now 1790545143360351 us; error 85 us; skew 500 ppm
I20260927 21:39:03.360863 1715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:03.364539 1715 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:03.366307 1715 webserver.cc:460] Webserver started at http://127.0.202.131:35419/ using document root <none> and password file <none>
I20260927 21:39:03.367110 1715 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:03.367274 1715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:03.367622 1715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:03.370977 1715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "986ef1fabd3246efa081d2dc75ddfe84"
format_stamp: "Formatted at 2026-09-27 21:39:03 on dist-test-slave-25dq"
server_key: "47c59348156f3fc986841362f7b28d0e"
server_key_iv: "fe671564faf7c85f3013a3d57eae5625"
server_key_version: "encryptionkey@0"
I20260927 21:39:03.371902 1715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "986ef1fabd3246efa081d2dc75ddfe84"
format_stamp: "Formatted at 2026-09-27 21:39:03 on dist-test-slave-25dq"
server_key: "47c59348156f3fc986841362f7b28d0e"
server_key_iv: "fe671564faf7c85f3013a3d57eae5625"
server_key_version: "encryptionkey@0"
I20260927 21:39:03.377933 1715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260927 21:39:03.381824 1730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:03.383445 1715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:03.383656 1715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "986ef1fabd3246efa081d2dc75ddfe84"
format_stamp: "Formatted at 2026-09-27 21:39:03 on dist-test-slave-25dq"
server_key: "47c59348156f3fc986841362f7b28d0e"
server_key_iv: "fe671564faf7c85f3013a3d57eae5625"
server_key_version: "encryptionkey@0"
I20260927 21:39:03.383878 1715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:03.405628 1715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:03.410283 1715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:03.410598 1715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:03.411644 1715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:03.413141 1715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:03.413242 1715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:03.413388 1715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:03.413493 1715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:03.435107 1575 heartbeater.cc:503] Master 127.0.202.190:41359 was elected leader, sending a full tablet report...
I20260927 21:39:03.448391 1715 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:42729
I20260927 21:39:03.448433 1842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:42729 every 8 connection(s)
I20260927 21:39:03.449705 1715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:03.452329 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 1715
I20260927 21:39:03.452574 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:03.453013 810 external_mini_cluster.cc:1428] Setting key 6defb9623f4515e3acae3948dd98a724
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:03.471503 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.452561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:36421 (local address 127.0.202.190:41359)
0927 21:39:03.452768 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.452775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.454147 (+ 1372us) server_negotiation.cc:407] Connection header received
0927 21:39:03.456510 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.456516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.456615 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.456745 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:03.458240 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.459369 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.460989 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.461381 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.461519 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.465033 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:03.465039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:03.465042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:03.465079 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:03.468152 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.469147 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.469152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.469155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.469298 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.469805 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.469809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.469811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.470118 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:03.470345 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.471027 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.471212 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:39:03.472586 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.453153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41359 (local address 127.0.202.131:36421)
0927 21:39:03.453967 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:03.454022 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:03.456197 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:03.457001 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:03.457010 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:03.457514 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:03.458079 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.458095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.459517 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.459522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.460852 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.460863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.461601 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.461608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.461836 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.462501 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:03.462526 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:03.464803 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:03.468326 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:03.468338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:03.468358 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:03.468999 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:03.469455 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:03.469459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:03.469462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:03.469652 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:03.470263 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:03.470271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:03.470622 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 21:39:03.470897 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 21:39:03.471305 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":328,"threads_started":1}
I20260927 21:39:03.474522 1843 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41359
I20260927 21:39:03.474876 1843 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:03.475698 1843 heartbeater.cc:511] Master 127.0.202.190:41359 requested a full tablet report, sending...
I20260927 21:39:03.477423 1386 ts_manager.cc:194] Registered new tserver with Master: 986ef1fabd3246efa081d2dc75ddfe84 (127.0.202.131:42729)
I20260927 21:39:03.478255 1386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:36421
I20260927 21:39:03.481421 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:39:03.501940 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.488056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.0.202.190:41359)
0927 21:39:03.488368 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.488383 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.488727 (+ 344us) server_negotiation.cc:407] Connection header received
0927 21:39:03.488863 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.488867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.488973 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.489073 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:03.490574 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.491439 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.492927 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.493370 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.493481 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.495962 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:03.495966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:03.495969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:03.496004 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:03.499030 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.499885 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.499893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.499898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.500077 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.500551 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.500556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.500560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.500899 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:03.501090 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.501485 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.501654 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:39:03.509497 1386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:03.517295 1386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:03.543716 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.530582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.0.202.131:42729)
0927 21:39:03.531854 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.531868 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.533786 (+ 1918us) server_negotiation.cc:407] Connection header received
0927 21:39:03.534257 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.534267 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.534576 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.534759 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:03.535890 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.537647 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.540454 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.542094 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.542228 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.542503 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.543147 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.543349 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":807,"threads_started":1}
I20260927 21:39:03.543807 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.530490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42729 (local address 127.0.0.1:49542)
0927 21:39:03.533635 (+ 3145us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:03.533673 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:03.533965 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:03.535147 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:03.535159 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:03.535197 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:03.535655 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.535673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.537869 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.537875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.540253 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.540268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.542642 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.542649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.542853 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.542910 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:39:03.543065 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 21:39:03.543313 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2888,"mutex_wait_us":55,"spinlock_wait_cycles":896,"thread_start_us":2217,"threads_started":1}
I20260927 21:39:03.544602 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.532033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:44439 (local address 127.0.0.1:60412)
0927 21:39:03.535079 (+ 3046us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:03.535132 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:03.535252 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:03.535934 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:03.535951 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:03.535979 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:03.536459 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.536475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.538549 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.538556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.540647 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.540661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.543270 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.543278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.543483 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.543525 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:39:03.543611 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:39:03.543771 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2906,"mutex_wait_us":34,"thread_start_us":2652,"threads_started":1}
I20260927 21:39:03.544981 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.531369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.0.202.129:44439)
0927 21:39:03.533254 (+ 1885us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.533261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.535404 (+ 2143us) server_negotiation.cc:407] Connection header received
0927 21:39:03.535513 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.535522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.535752 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.535950 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:03.536645 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.538370 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.540798 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.542734 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.542879 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.543122 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.543931 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.544126 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1691,"thread_start_us":1534,"threads_started":1}
I20260927 21:39:03.545799 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.531626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33951 (local address 127.0.0.1:42282)
0927 21:39:03.534362 (+ 2736us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:03.534380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:03.534491 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:03.535510 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:03.535515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:03.535546 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:03.536145 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.536164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.537824 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.537833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.541935 (+ 4102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.541950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.544894 (+ 2944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.544905 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.545083 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.545118 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:39:03.545266 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 21:39:03.545417 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2605,"mutex_wait_us":1534,"thread_start_us":2339,"threads_started":1}
I20260927 21:39:03.546265 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.531320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.0.202.130:33951)
0927 21:39:03.532056 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.532067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.534940 (+ 2873us) server_negotiation.cc:407] Connection header received
0927 21:39:03.535047 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.535057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.535312 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.535526 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:03.536375 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.537659 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.542143 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.543742 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.544373 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.544559 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.545341 (+ 782us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.545516 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":452,"threads_started":1}
I20260927 21:39:03.548622 1778 tablet_service.cc:1467] Processing CreateTablet for tablet eb8efe627b6341418cea6a9fb10456bf (DEFAULT_TABLE table=test-table [id=60279b77e5af4b469b62da1626a7cdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:03.548677 1510 tablet_service.cc:1467] Processing CreateTablet for tablet eb8efe627b6341418cea6a9fb10456bf (DEFAULT_TABLE table=test-table [id=60279b77e5af4b469b62da1626a7cdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:03.550069 1778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8efe627b6341418cea6a9fb10456bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:03.550158 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8efe627b6341418cea6a9fb10456bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:03.550195 1644 tablet_service.cc:1467] Processing CreateTablet for tablet eb8efe627b6341418cea6a9fb10456bf (DEFAULT_TABLE table=test-table [id=60279b77e5af4b469b62da1626a7cdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:03.551468 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8efe627b6341418cea6a9fb10456bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:03.563077 1861 tablet_bootstrap.cc:492] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84: Bootstrap starting.
I20260927 21:39:03.566720 1861 tablet_bootstrap.cc:654] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:03.568010 1863 tablet_bootstrap.cc:492] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17: Bootstrap starting.
I20260927 21:39:03.568348 1861 log.cc:824] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:03.569916 1862 tablet_bootstrap.cc:492] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2: Bootstrap starting.
I20260927 21:39:03.572749 1863 tablet_bootstrap.cc:654] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:03.573582 1862 tablet_bootstrap.cc:654] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:03.574096 1863 log.cc:824] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:03.574657 1862 log.cc:824] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:03.575982 1861 tablet_bootstrap.cc:492] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84: No bootstrap required, opened a new log
I20260927 21:39:03.576390 1861 ts_tablet_manager.cc:1397] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260927 21:39:03.579233 1863 tablet_bootstrap.cc:492] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17: No bootstrap required, opened a new log
I20260927 21:39:03.579516 1863 ts_tablet_manager.cc:1397] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 21:39:03.581632 1862 tablet_bootstrap.cc:492] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2: No bootstrap required, opened a new log
I20260927 21:39:03.581912 1862 ts_tablet_manager.cc:1397] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260927 21:39:03.588420 1861 raft_consensus.cc:348] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.588918 1861 raft_consensus.cc:374] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:03.589022 1861 raft_consensus.cc:729] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986ef1fabd3246efa081d2dc75ddfe84, State: Initialized, Role: FOLLOWER
I20260927 21:39:03.589701 1861 consensus_queue.cc:260] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.591506 1862 raft_consensus.cc:348] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.591903 1862 raft_consensus.cc:374] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:03.592029 1862 raft_consensus.cc:729] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d324728ce94428ad7e5f74a2b0c4e2, State: Initialized, Role: FOLLOWER
I20260927 21:39:03.592712 1863 raft_consensus.cc:348] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.592890 1862 consensus_queue.cc:260] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.593094 1843 heartbeater.cc:503] Master 127.0.202.190:41359 was elected leader, sending a full tablet report...
I20260927 21:39:03.593142 1863 raft_consensus.cc:374] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:03.593232 1863 raft_consensus.cc:729] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52437120dc0246e0a9eb75a9d69e3d17, State: Initialized, Role: FOLLOWER
I20260927 21:39:03.593853 1863 consensus_queue.cc:260] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.594126 1861 ts_tablet_manager.cc:1428] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260927 21:39:03.595068 1709 heartbeater.cc:503] Master 127.0.202.190:41359 was elected leader, sending a full tablet report...
I20260927 21:39:03.595808 1863 ts_tablet_manager.cc:1428] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260927 21:39:03.597684 1862 ts_tablet_manager.cc:1428] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260927 21:39:03.608107 1867 raft_consensus.cc:482] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:03.608552 1867 raft_consensus.cc:504] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.610220 1867 leader_election.cc:288] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4d324728ce94428ad7e5f74a2b0c4e2 (127.0.202.129:44439), 52437120dc0246e0a9eb75a9d69e3d17 (127.0.202.130:33951)
I20260927 21:39:03.617522 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.610904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:48043 (local address 127.0.202.129:44439)
0927 21:39:03.611118 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.611127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.611264 (+ 137us) server_negotiation.cc:407] Connection header received
0927 21:39:03.611515 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.611520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.611616 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.611748 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:03.612443 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.613477 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.615422 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.616643 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.616809 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.616893 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.617185 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.617333 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:39:03.617633 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.610883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:44439 (local address 127.0.202.131:48043)
0927 21:39:03.611127 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:03.611153 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:03.611351 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:03.611796 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:03.611803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:03.611839 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:03.612123 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.612135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.613633 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.613639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.615222 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.615234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.616790 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.616795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.617020 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.617047 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 21:39:03.617137 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:39:03.617240 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 21:39:03.618806 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8efe627b6341418cea6a9fb10456bf" candidate_uuid: "986ef1fabd3246efa081d2dc75ddfe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" is_pre_election: true
I20260927 21:39:03.619501 1530 raft_consensus.cc:2454] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986ef1fabd3246efa081d2dc75ddfe84 in term 0.
I20260927 21:39:03.619802 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.611377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33951 (local address 127.0.202.131:53925)
0927 21:39:03.612246 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:03.612264 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:03.612406 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:03.612820 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:03.612823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:03.612840 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:03.613729 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.613740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.615118 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.615124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.617187 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:03.617201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.619063 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.619068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:03.619227 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.619259 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:39:03.619361 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:39:03.619508 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":447,"threads_started":1}
I20260927 21:39:03.619858 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.610966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:53925 (local address 127.0.202.130:33951)
0927 21:39:03.611192 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.611201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.612457 (+ 1256us) server_negotiation.cc:407] Connection header received
0927 21:39:03.612602 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.612608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.612689 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.612793 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:03.613905 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.614949 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.617354 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.618695 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.618870 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.618951 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.619419 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.619581 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:39:03.620393 1733 leader_election.cc:302] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 986ef1fabd3246efa081d2dc75ddfe84, e4d324728ce94428ad7e5f74a2b0c4e2; no voters:
I20260927 21:39:03.620888 1867 raft_consensus.cc:2781] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:03.621054 1867 raft_consensus.cc:482] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:03.621171 1867 raft_consensus.cc:3037] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:03.620961 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8efe627b6341418cea6a9fb10456bf" candidate_uuid: "986ef1fabd3246efa081d2dc75ddfe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52437120dc0246e0a9eb75a9d69e3d17" is_pre_election: true
I20260927 21:39:03.621563 1664 raft_consensus.cc:2454] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986ef1fabd3246efa081d2dc75ddfe84 in term 0.
I20260927 21:39:03.623860 1867 raft_consensus.cc:504] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.624562 1867 leader_election.cc:288] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [CANDIDATE]: Term 1 election: Requested vote from peers e4d324728ce94428ad7e5f74a2b0c4e2 (127.0.202.129:44439), 52437120dc0246e0a9eb75a9d69e3d17 (127.0.202.130:33951)
I20260927 21:39:03.625149 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8efe627b6341418cea6a9fb10456bf" candidate_uuid: "986ef1fabd3246efa081d2dc75ddfe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52437120dc0246e0a9eb75a9d69e3d17"
I20260927 21:39:03.625370 1664 raft_consensus.cc:3037] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:03.625590 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8efe627b6341418cea6a9fb10456bf" candidate_uuid: "986ef1fabd3246efa081d2dc75ddfe84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2"
I20260927 21:39:03.625816 1530 raft_consensus.cc:3037] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:03.627985 1530 raft_consensus.cc:2454] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986ef1fabd3246efa081d2dc75ddfe84 in term 1.
I20260927 21:39:03.628219 1664 raft_consensus.cc:2454] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986ef1fabd3246efa081d2dc75ddfe84 in term 1.
I20260927 21:39:03.628614 1733 leader_election.cc:302] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 986ef1fabd3246efa081d2dc75ddfe84, e4d324728ce94428ad7e5f74a2b0c4e2; no voters:
I20260927 21:39:03.629029 1867 raft_consensus.cc:2781] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:03.629460 1867 raft_consensus.cc:686] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [term 1 LEADER]: Becoming Leader. State: Replica: 986ef1fabd3246efa081d2dc75ddfe84, State: Running, Role: LEADER
I20260927 21:39:03.630034 1867 consensus_queue.cc:237] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } }
I20260927 21:39:03.635295 1385 catalog_manager.cc:5495] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 986ef1fabd3246efa081d2dc75ddfe84 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "986ef1fabd3246efa081d2dc75ddfe84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986ef1fabd3246efa081d2dc75ddfe84" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d324728ce94428ad7e5f74a2b0c4e2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 44439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52437120dc0246e0a9eb75a9d69e3d17" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33951 } health_report { overall_health: UNKNOWN } } }
W20260927 21:39:03.653163 1576 tablet.cc:2369] T eb8efe627b6341418cea6a9fb10456bf P e4d324728ce94428ad7e5f74a2b0c4e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:03.679698 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.662726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.0.202.129:44439)
0927 21:39:03.663009 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.663019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.663390 (+ 371us) server_negotiation.cc:407] Connection header received
0927 21:39:03.663535 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.663539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.663602 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.663698 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:03.665923 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.666703 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.668047 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.668546 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.668645 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.673244 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:03.673268 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:03.673292 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:03.673345 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:03.676819 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.677616 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.677625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.677627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.677763 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.678159 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.678163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.678165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.678568 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:03.678806 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.679064 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.679234 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 21:39:03.683003 1710 tablet.cc:2369] T eb8efe627b6341418cea6a9fb10456bf P 52437120dc0246e0a9eb75a9d69e3d17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:39:03.701484 1844 tablet.cc:2369] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:03.731806 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.716428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.0.202.129:44439)
0927 21:39:03.716727 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.716734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.717101 (+ 367us) server_negotiation.cc:407] Connection header received
0927 21:39:03.717220 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.717225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.717295 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.717408 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:03.719037 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.719870 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.721210 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.721713 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.721864 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.726028 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:03.726033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:03.726037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:03.726075 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:03.729273 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.729909 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.729914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.729916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.730041 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.730463 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.730466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.730468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.730740 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:03.730934 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.731339 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.731524 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:03.780699 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.763632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.202.129:44439)
0927 21:39:03.763855 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.763875 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.764211 (+ 336us) server_negotiation.cc:407] Connection header received
0927 21:39:03.764503 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.764507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.764577 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.764692 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:03.766159 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.766929 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.768738 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.769219 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.769362 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.774300 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:03.774307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:03.774312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:03.774358 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:03.777551 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.778371 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.778386 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.778390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.778522 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.778976 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.778987 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.778990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.779355 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:03.779638 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.779993 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.780248 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 21:39:03.782787 1490 tablet_service.cc:2908] Rejecting scan request for tablet eb8efe627b6341418cea6a9fb10456bf: Uninitialized: clean time has not yet been initialized
I20260927 21:39:03.786118 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 1447
I20260927 21:39:03.801786 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.787080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.0.202.129:44439)
0927 21:39:03.787296 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.787307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.787856 (+ 549us) server_negotiation.cc:407] Connection header received
0927 21:39:03.788067 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.788073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.788156 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.788272 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:03.789601 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.790339 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.791649 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.792087 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.792243 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.794946 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:03.794952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:03.794956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:03.794992 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:03.798069 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.798927 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.798936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.798946 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.799080 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.799989 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.799997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.800001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.800433 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:03.800755 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.801125 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.801394 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:39:03.802564 1570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:03.894246 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 1447
I20260927 21:39:03.913905 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 1581
Sep 27 21:39:03 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:39:03.930179 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:03.914737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.0.202.130:33951)
0927 21:39:03.914937 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:03.914942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:03.915000 (+ 58us) server_negotiation.cc:407] Connection header received
0927 21:39:03.915137 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:03.915149 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:03.915221 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:03.915315 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:03.916595 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.917431 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.918791 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:03.919478 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:03.919590 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:03.924127 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:03.924140 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:03.924156 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:03.924194 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:03.927042 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.927745 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.927752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.927755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.927898 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:03.928375 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:03.928379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:03.928382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:03.928911 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:03.929149 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:03.929578 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 21:39:03.929754 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 21:39:03.930953 1704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:04.030117 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 1581
I20260927 21:39:04.044008 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 1715
I20260927 21:39:04.046561 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:04.046041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33951 (local address 127.0.202.131:42125)
0927 21:39:04.046222 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:04.046368 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 21:39:04.046890 1732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33951: connect: Connection refused (error 111)
I20260927 21:39:04.048501 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:04.048111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33951 (local address 127.0.202.131:58169)
0927 21:39:04.048237 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:04.048357 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 21:39:04.048702 1732 consensus_peers.cc:487] T eb8efe627b6341418cea6a9fb10456bf P 986ef1fabd3246efa081d2dc75ddfe84 -> Peer 52437120dc0246e0a9eb75a9d69e3d17 (127.0.202.130:33951): Couldn't send request to peer 52437120dc0246e0a9eb75a9d69e3d17. Status: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33951: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 27 21:39:04 dist-test-slave-25dq krb5kdc[1341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:39:04.061321 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:04.044957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.0.202.131:42729)
0927 21:39:04.045387 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:04.045391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:04.045408 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:39:04.045460 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:04.045463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:04.045520 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:04.045610 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:04.047200 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:04.048462 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:04.049832 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:04.050254 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:04.050359 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:04.054913 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:04.054927 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:04.054943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:04.054987 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:04.058065 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:04.059026 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:04.059035 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:04.059038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:04.059181 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:04.059647 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:04.059652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:04.059654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:04.060093 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:04.060542 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:04.060827 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 21:39:04.061007 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":233,"threads_started":1}
I20260927 21:39:04.062240 1838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:04.153465 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:04.153039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:44439 (local address 127.0.202.131:34755)
0927 21:39:04.153198 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:04.153290 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 21:39:04.153865 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 1715
I20260927 21:39:04.170729 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 1354
I20260927 21:39:04.171933 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:04.283978 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 1354
2026-09-27T21:39:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4322 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:04 dist-test-slave-25dq krb5kdc[1892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:04 dist-test-slave-25dq krb5kdc[1892](info): set up 2 sockets
Sep 27 21:39:04 dist-test-slave-25dq krb5kdc[1892](info): commencing operation
krb5kdc: starting...
W20260927 21:39:05.416281 810 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:05 dist-test-slave-25dq krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:05.612939 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:38973
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:05.942020 1905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:05.942338 1905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:05.942430 1905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:05.949752 1905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:05.949868 1905 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:39:05.949909 1905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:05.949962 1905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:05.950011 1905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:05.950073 1905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:05.960055 1905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:38973
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:05.961410 1905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:05.962798 1905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:05.975198 1910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:05.975198 1911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:05.975206 1913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:05.976019 1905 server_base.cc:1034] running on GCE node
I20260927 21:39:05.976915 1905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:05.978518 1905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:05.979852 1905 hybrid_clock.cc:648] HybridClock initialized: now 1790545145979738 us; error 47 us; skew 500 ppm
I20260927 21:39:05.980278 1905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:05 dist-test-slave-25dq krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:05.983791 1905 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:05.985711 1905 webserver.cc:460] Webserver started at http://127.0.202.190:42239/ using document root <none> and password file <none>
I20260927 21:39:05.986518 1905 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:05.986618 1905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:05.987030 1905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:05.990628 1905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "b5d82af54d1f4916afcac3811f064fde"
format_stamp: "Formatted at 2026-09-27 21:39:05 on dist-test-slave-25dq"
server_key: "9c3731ff7358e9a519288dee5b67cfe2"
server_key_iv: "a22f8129c23266c6dc820132787fcdca"
server_key_version: "encryptionkey@0"
I20260927 21:39:05.991490 1905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "b5d82af54d1f4916afcac3811f064fde"
format_stamp: "Formatted at 2026-09-27 21:39:05 on dist-test-slave-25dq"
server_key: "9c3731ff7358e9a519288dee5b67cfe2"
server_key_iv: "a22f8129c23266c6dc820132787fcdca"
server_key_version: "encryptionkey@0"
I20260927 21:39:05.996934 1905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:39:06.000613 1920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:06.002022 1905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:06.002194 1905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "b5d82af54d1f4916afcac3811f064fde"
format_stamp: "Formatted at 2026-09-27 21:39:05 on dist-test-slave-25dq"
server_key: "9c3731ff7358e9a519288dee5b67cfe2"
server_key_iv: "a22f8129c23266c6dc820132787fcdca"
server_key_version: "encryptionkey@0"
I20260927 21:39:06.002415 1905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:06.026316 1905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:06.031298 1905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:06.031559 1905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:06.056254 1905 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:38973
I20260927 21:39:06.056325 1981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:38973 every 8 connection(s)
I20260927 21:39:06.057713 1905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:06.061729 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:06.061789 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 1905
I20260927 21:39:06.062040 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:06.062841 810 external_mini_cluster.cc:1428] Setting key b61d1bd55972c38f3302a7c4714de5c8
I20260927 21:39:06.077768 1982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde: Bootstrap starting.
Sep 27 21:39:06 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:06.081781 1982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:06.083318 1982 log.cc:824] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:06.087160 1982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde: No bootstrap required, opened a new log
I20260927 21:39:06.091234 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:06.065339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.0.202.190:38973)
0927 21:39:06.066230 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:06.066253 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:06.066308 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:39:06.069095 (+ 2787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:06.069131 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:06.069787 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:06.070318 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:06.071734 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.073918 (+ 2184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.075285 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.076462 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.076665 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:06.081429 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:06.081440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:06.081463 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:06.081523 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:06.085977 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:06.086875 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:06.086884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:06.086893 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:06.087113 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:06.087617 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:06.087624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:06.087628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:06.088410 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:06.088709 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:06.089241 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 21:39:06.089728 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":364,"threads_started":1}
I20260927 21:39:06.096036 1982 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d82af54d1f4916afcac3811f064fde" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 38973 } }
I20260927 21:39:06.096467 1982 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:06.096547 1982 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d82af54d1f4916afcac3811f064fde, State: Initialized, Role: FOLLOWER
I20260927 21:39:06.097121 1982 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d82af54d1f4916afcac3811f064fde" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 38973 } }
I20260927 21:39:06.097374 1982 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:06.097532 1982 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:06.097714 1982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:06.099840 1982 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d82af54d1f4916afcac3811f064fde" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 38973 } }
I20260927 21:39:06.100433 1982 leader_election.cc:302] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d82af54d1f4916afcac3811f064fde; no voters:
I20260927 21:39:06.101068 1982 leader_election.cc:288] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:06.101208 1987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:06.101940 1987 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [term 1 LEADER]: Becoming Leader. State: Replica: b5d82af54d1f4916afcac3811f064fde, State: Running, Role: LEADER
I20260927 21:39:06.102576 1987 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d82af54d1f4916afcac3811f064fde" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 38973 } }
I20260927 21:39:06.103210 1982 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:06.106050 1988 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d82af54d1f4916afcac3811f064fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d82af54d1f4916afcac3811f064fde" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 38973 } } }
I20260927 21:39:06.106403 1988 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:06.106999 1989 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d82af54d1f4916afcac3811f064fde. Latest consensus state: current_term: 1 leader_uuid: "b5d82af54d1f4916afcac3811f064fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d82af54d1f4916afcac3811f064fde" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 38973 } } }
I20260927 21:39:06.107389 1989 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:06.109426 1994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:06.114517 1994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:06.123787 1994 catalog_manager.cc:1347] Generated new cluster ID: ad754f873f76497098b4c8f0607d1e67
I20260927 21:39:06.123921 1994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:06.141875 1994 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:06.143558 1994 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:06.149585 1994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde: Generated new TSK 0
I20260927 21:39:06.150545 1994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:06.201270 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:38973
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:06.551470 2008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:06.551782 2008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:06.551885 2008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:06.559443 2008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:06.559554 2008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:06.559638 2008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:39:06.559717 2008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:06.569679 2008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:06.571118 2008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:06.572461 2008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:06.586603 2013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:06.586570 2014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:06.586549 2016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:06.587514 2008 server_base.cc:1034] running on GCE node
I20260927 21:39:06.588147 2008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:06.589195 2008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:06.590559 2008 hybrid_clock.cc:648] HybridClock initialized: now 1790545146590388 us; error 67 us; skew 500 ppm
I20260927 21:39:06.590930 2008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:06 dist-test-slave-25dq krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:06.594453 2008 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:06.595960 2008 webserver.cc:460] Webserver started at http://127.0.202.129:36319/ using document root <none> and password file <none>
I20260927 21:39:06.596710 2008 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:06.596801 2008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:06.597074 2008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:06.599944 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "2e095308069841bd82e664bcc38dc536"
format_stamp: "Formatted at 2026-09-27 21:39:06 on dist-test-slave-25dq"
server_key: "a3d8c75043ea5035e5786607394e718f"
server_key_iv: "9e4946e15b1487d825b71b8d7fc87d63"
server_key_version: "encryptionkey@0"
I20260927 21:39:06.600736 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "2e095308069841bd82e664bcc38dc536"
format_stamp: "Formatted at 2026-09-27 21:39:06 on dist-test-slave-25dq"
server_key: "a3d8c75043ea5035e5786607394e718f"
server_key_iv: "9e4946e15b1487d825b71b8d7fc87d63"
server_key_version: "encryptionkey@0"
I20260927 21:39:06.606108 2008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:39:06.610163 2023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:06.611624 2008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:39:06.611898 2008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "2e095308069841bd82e664bcc38dc536"
format_stamp: "Formatted at 2026-09-27 21:39:06 on dist-test-slave-25dq"
server_key: "a3d8c75043ea5035e5786607394e718f"
server_key_iv: "9e4946e15b1487d825b71b8d7fc87d63"
server_key_version: "encryptionkey@0"
I20260927 21:39:06.612112 2008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:06.646544 2008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:06.652163 2008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:06.652540 2008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:06.655616 2008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:06.655709 2008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:06.655865 2008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:06.655967 2008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:39:06 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:06.680936 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:06.656886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:48743 (local address 127.0.202.190:38973)
0927 21:39:06.660691 (+ 3805us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:06.660698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:06.660733 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:39:06.661058 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:06.661064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:06.661149 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:06.661279 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:06.663507 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.664698 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.667775 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.668340 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.668465 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:06.674032 (+ 5567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:06.674037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:06.674041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:06.674079 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:06.677126 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:06.678273 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:06.678281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:06.678284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:06.678426 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:06.678932 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:06.678943 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:06.678963 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:06.679243 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:06.679407 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:06.680239 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 21:39:06.680520 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3669,"thread_start_us":3559,"threads_started":1}
I20260927 21:39:06.682726 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:06.657429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:38973 (local address 127.0.202.129:48743)
0927 21:39:06.658362 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:06.658420 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:06.660631 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:06.661536 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:06.661545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:06.662280 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:06.663275 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:06.663304 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.664901 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.664905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:06.667519 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:06.667532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.668616 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.668628 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:06.669045 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:06.670079 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:06.670125 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:06.673720 (+ 3595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:06.677291 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:06.677305 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:06.677324 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:06.678112 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:06.678556 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:06.678562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:06.678566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:06.678767 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:06.679406 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:06.679417 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:06.679810 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 21:39:06.680110 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 21:39:06.680771 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":404,"threads_started":1}
I20260927 21:39:06.710153 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:06.689189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:40235 (local address 127.0.202.190:38973)
0927 21:39:06.689378 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:06.689388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:06.691186 (+ 1798us) server_negotiation.cc:407] Connection header received
0927 21:39:06.691403 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:06.691410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:06.691485 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:06.691601 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:06.692704 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.693417 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.694776 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.695198 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.695341 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:06.703340 (+ 7999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:06.703348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:06.703353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:06.703400 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:06.707034 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:06.707789 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:06.707795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:06.707798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:06.707968 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:06.708404 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:06.708410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:06.708414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:06.708727 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:06.708903 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:06.709543 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 21:39:06.709764 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:39:06.710266 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:06.689224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:38973 (local address 127.0.202.129:40235)
0927 21:39:06.691061 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:06.691084 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:06.691254 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:06.691623 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:06.691627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:06.692104 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:06.692533 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:06.692546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.693572 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.693577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:06.694632 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:06.694644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:06.695388 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:06.695394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:06.695575 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:06.701443 (+ 5868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:06.701465 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:06.702945 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:06.707203 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:06.707208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:06.707212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:06.707648 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:06.708093 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:06.708099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:06.708102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:06.708237 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:06.708882 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:06.708888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:06.709249 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 21:39:06.709428 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 21:39:06.709788 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1591,"thread_start_us":955,"threads_started":1}
I20260927 21:39:06.720925 2008 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:39613
I20260927 21:39:06.720959 2143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:39613 every 8 connection(s)
I20260927 21:39:06.722283 2008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:06.724704 2144 heartbeater.cc:347] Connected to a master server at 127.0.202.190:38973
I20260927 21:39:06.724998 2144 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:06.725693 2144 heartbeater.cc:511] Master 127.0.202.190:38973 requested a full tablet report, sending...
I20260927 21:39:06.727851 1937 ts_manager.cc:194] Registered new tserver with Master: 2e095308069841bd82e664bcc38dc536 (127.0.202.129:39613)
I20260927 21:39:06.729420 1937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:48743
I20260927 21:39:06.732800 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 2008
I20260927 21:39:06.733006 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:06.733646 810 external_mini_cluster.cc:1428] Setting key 89f2ed7a69c07a1fcf524c2d13645ba5
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:06.776844 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:38973
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:07.111938 2148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:07.112246 2148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:07.112428 2148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:07.120136 2148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:07.120265 2148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:07.120447 2148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:39:07.120605 2148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:07.131080 2148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:07.132436 2148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:07.134027 2148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:07.147634 2156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:07.147713 2154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:07.147668 2153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:07.148907 2148 server_base.cc:1034] running on GCE node
I20260927 21:39:07.149631 2148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:07.150720 2148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:07.151996 2148 hybrid_clock.cc:648] HybridClock initialized: now 1790545147151933 us; error 38 us; skew 500 ppm
I20260927 21:39:07.152406 2148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:07 dist-test-slave-25dq krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:07.155813 2148 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:07.157510 2148 webserver.cc:460] Webserver started at http://127.0.202.130:39287/ using document root <none> and password file <none>
I20260927 21:39:07.158201 2148 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:07.158296 2148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:07.158591 2148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:07.161620 2148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "6475cc3fabfa456a9ebc67bf01cf9376"
format_stamp: "Formatted at 2026-09-27 21:39:07 on dist-test-slave-25dq"
server_key: "23d800d0e03f5a861526236e1625fa8b"
server_key_iv: "b0d9792d30f3eb3804ef61603e4648ce"
server_key_version: "encryptionkey@0"
I20260927 21:39:07.162410 2148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "6475cc3fabfa456a9ebc67bf01cf9376"
format_stamp: "Formatted at 2026-09-27 21:39:07 on dist-test-slave-25dq"
server_key: "23d800d0e03f5a861526236e1625fa8b"
server_key_iv: "b0d9792d30f3eb3804ef61603e4648ce"
server_key_version: "encryptionkey@0"
I20260927 21:39:07.168041 2148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:39:07.171905 2163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:07.173524 2148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:07.173708 2148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "6475cc3fabfa456a9ebc67bf01cf9376"
format_stamp: "Formatted at 2026-09-27 21:39:07 on dist-test-slave-25dq"
server_key: "23d800d0e03f5a861526236e1625fa8b"
server_key_iv: "b0d9792d30f3eb3804ef61603e4648ce"
server_key_version: "encryptionkey@0"
I20260927 21:39:07.173950 2148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:07.195605 2148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:07.200850 2148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:07.201179 2148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:07.204413 2148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:07.204538 2148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:07.204704 2148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:07.204771 2148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:39:07 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:07.230175 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.205770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:41081 (local address 127.0.202.190:38973)
0927 21:39:07.206010 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.206018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.207588 (+ 1570us) server_negotiation.cc:407] Connection header received
0927 21:39:07.210473 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.210479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.210579 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.210717 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:07.212391 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.213549 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.216692 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.217433 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.217585 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.222725 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:07.222730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:07.222734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:07.222770 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:07.226696 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.227576 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.227584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.227590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.227779 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.228356 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.228361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.228365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.228653 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:07.228819 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.229614 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.229830 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:39:07.231616 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.206250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:38973 (local address 127.0.202.130:41081)
0927 21:39:07.207359 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.207431 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.210085 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.210980 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.210988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.211554 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:07.212174 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.212193 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.213790 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.213794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.216490 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.216506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.217718 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.217727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.217979 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.218811 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:07.218841 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:07.222446 (+ 3605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:07.226855 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.226864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.226887 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.227394 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.227923 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.227931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.227937 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.228184 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.228788 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:07.228797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:07.229200 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 21:39:07.229496 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 21:39:07.230050 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":479,"threads_started":1}
I20260927 21:39:07.257176 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.240189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:57219 (local address 127.0.202.190:38973)
0927 21:39:07.240504 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.240515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.241125 (+ 610us) server_negotiation.cc:407] Connection header received
0927 21:39:07.241307 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.241316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.241436 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.241594 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:07.242577 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.244015 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.246338 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.247360 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.247562 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.250117 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:07.250126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:07.250132 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:07.250176 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:07.253871 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.254823 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.254830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.254836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.255024 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.255470 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.255475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.255478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.255951 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:07.256113 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.256587 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.256817 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:39:07.257339 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.240218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:38973 (local address 127.0.202.130:57219)
0927 21:39:07.240984 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.241013 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.241163 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.241671 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.241675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.242115 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:07.242407 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.242417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.244229 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.244237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.246146 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.246162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.247608 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.247618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.247774 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.248675 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:07.248708 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:07.249857 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:07.254024 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.254031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.254036 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.254681 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.255179 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.255186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.255193 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.255342 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.256076 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:07.256082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:07.256270 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 21:39:07.256473 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 21:39:07.256844 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":422,"threads_started":1}
I20260927 21:39:07.265748 2148 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:39433
I20260927 21:39:07.265765 2282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:39433 every 8 connection(s)
I20260927 21:39:07.267131 2148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:07.269400 2283 heartbeater.cc:347] Connected to a master server at 127.0.202.190:38973
I20260927 21:39:07.269728 2283 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:07.270431 2283 heartbeater.cc:511] Master 127.0.202.190:38973 requested a full tablet report, sending...
I20260927 21:39:07.271833 1937 ts_manager.cc:194] Registered new tserver with Master: 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130:39433)
I20260927 21:39:07.272872 1937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:41081
I20260927 21:39:07.276932 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 2148
I20260927 21:39:07.277173 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:07.277726 810 external_mini_cluster.cc:1428] Setting key 09f22afaca1570ac3f0c09443c0fd0a1
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:07.323850 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:38973
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:07.660308 2287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:07.660615 2287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:07.660794 2287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:07.668789 2287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:07.668965 2287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:07.669126 2287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:39:07.669253 2287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:07.679414 2287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:07.680827 2287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:07.682155 2287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:07.695766 2295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:07.695927 2292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:07.695850 2293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:07.697409 2287 server_base.cc:1034] running on GCE node
I20260927 21:39:07.698223 2287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:07.699347 2287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:07.700819 2287 hybrid_clock.cc:648] HybridClock initialized: now 1790545147700708 us; error 87 us; skew 500 ppm
I20260927 21:39:07.701196 2287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:07 dist-test-slave-25dq krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:07.704955 2287 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:07.706872 2287 webserver.cc:460] Webserver started at http://127.0.202.131:34393/ using document root <none> and password file <none>
I20260927 21:39:07.707641 2287 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:07.707777 2287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:07.708112 2287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:07.711242 2287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "473d5fd7ad7a496fab21b586d1db400a"
format_stamp: "Formatted at 2026-09-27 21:39:07 on dist-test-slave-25dq"
server_key: "1c7273d95f8bd97c054837ce5f2e72ff"
server_key_iv: "b2aac34901e507e339aff0564b7b905b"
server_key_version: "encryptionkey@0"
I20260927 21:39:07.712072 2287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "473d5fd7ad7a496fab21b586d1db400a"
format_stamp: "Formatted at 2026-09-27 21:39:07 on dist-test-slave-25dq"
server_key: "1c7273d95f8bd97c054837ce5f2e72ff"
server_key_iv: "b2aac34901e507e339aff0564b7b905b"
server_key_version: "encryptionkey@0"
I20260927 21:39:07.717759 2287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:39:07.721524 2302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:07.723026 2287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:07.723232 2287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "473d5fd7ad7a496fab21b586d1db400a"
format_stamp: "Formatted at 2026-09-27 21:39:07 on dist-test-slave-25dq"
server_key: "1c7273d95f8bd97c054837ce5f2e72ff"
server_key_iv: "b2aac34901e507e339aff0564b7b905b"
server_key_version: "encryptionkey@0"
I20260927 21:39:07.723466 2287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:07.732568 2144 heartbeater.cc:503] Master 127.0.202.190:38973 was elected leader, sending a full tablet report...
I20260927 21:39:07.736672 2287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:07.740743 2287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:07.740962 2287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:07.743712 2287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:07.743815 2287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:07.743958 2287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:07.744040 2287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:39:07 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:07.768100 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.744380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:34273 (local address 127.0.202.190:38973)
0927 21:39:07.744598 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.744614 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.746052 (+ 1438us) server_negotiation.cc:407] Connection header received
0927 21:39:07.748151 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.748156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.748254 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.748410 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:07.751603 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.752965 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.755443 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.755867 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.756013 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.761341 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:07.761346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:07.761350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:07.761391 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:07.764064 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.765137 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.765144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.765149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.765287 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.765858 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.765866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.765869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.766228 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:07.766407 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.767480 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.767718 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:39:07.769381 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.745042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:38973 (local address 127.0.202.131:34273)
0927 21:39:07.745876 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.745916 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.747835 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.748753 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.748773 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.749806 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:07.751257 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.751294 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.753245 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.753250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.755256 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.755273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.756124 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.756131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.756411 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.757260 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:07.757283 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:07.761085 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:07.764217 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.764226 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.764247 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.764984 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.765408 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.765415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.765419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.765693 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.766510 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:07.766518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:07.767013 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 21:39:07.767358 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 21:39:07.767989 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":371,"threads_started":1}
I20260927 21:39:07.793110 2421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:42839 every 8 connection(s)
I20260927 21:39:07.793205 2287 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:42839
I20260927 21:39:07.794796 2287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:07.796813 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.777462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:38973 (local address 127.0.202.131:59225)
0927 21:39:07.778559 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.778582 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.778823 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.779329 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.779335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.780086 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:07.780758 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.780775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.782527 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.782534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.784598 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.784620 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.785921 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.785928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.786252 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.787277 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:07.787313 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:07.789190 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:07.793765 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.793770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.793773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.794245 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.794706 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:07.794710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:07.794712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:07.794890 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:07.795427 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:07.795432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:07.795700 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 21:39:07.795903 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 21:39:07.796271 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":766,"threads_started":1}
I20260927 21:39:07.796882 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.777811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:59225 (local address 127.0.202.190:38973)
0927 21:39:07.778083 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.778092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.778695 (+ 603us) server_negotiation.cc:407] Connection header received
0927 21:39:07.779034 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.779044 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.779149 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.779269 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:07.781042 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.782331 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.784813 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.785630 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.785800 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.789462 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:07.789471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:07.789479 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:07.789528 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:07.793579 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.794413 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.794418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.794422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.794570 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.795011 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.795016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.795020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.795290 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:07.795453 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.796059 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.796333 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:39:07.797569 2422 heartbeater.cc:347] Connected to a master server at 127.0.202.190:38973
I20260927 21:39:07.797830 2422 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:07.798658 2422 heartbeater.cc:511] Master 127.0.202.190:38973 requested a full tablet report, sending...
I20260927 21:39:07.800266 1937 ts_manager.cc:194] Registered new tserver with Master: 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131:42839)
I20260927 21:39:07.801188 1937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:34273
I20260927 21:39:07.803586 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 2287
I20260927 21:39:07.803797 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:07.804272 810 external_mini_cluster.cc:1428] Setting key 365859f375a1f3562f621de4750458d5
I20260927 21:39:07.807447 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:39:07 dist-test-slave-25dq krb5kdc[1892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:39:07 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:07.857306 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.841989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.0.202.190:38973)
0927 21:39:07.842242 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.842251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.842480 (+ 229us) server_negotiation.cc:407] Connection header received
0927 21:39:07.842707 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.842714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.842785 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.842895 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:07.844272 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.845079 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.846369 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.846826 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.847130 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.851614 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:07.851620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:07.851623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:07.851664 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:07.854563 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.855253 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.855265 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.855269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.855397 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:07.855845 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:07.855855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:07.855857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:07.856125 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:07.856397 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.856717 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.856935 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:39:07.861609 1937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:07.868595 1937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:07.893772 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.882353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.0.202.131:42839)
0927 21:39:07.883377 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.883387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.884060 (+ 673us) server_negotiation.cc:407] Connection header received
0927 21:39:07.884463 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.884471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.884723 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.884905 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:07.886177 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.887706 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.891045 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.892193 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.892362 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.892611 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.893284 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.893446 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":674,"threads_started":1}
I20260927 21:39:07.893817 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.881681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42839 (local address 127.0.0.1:46286)
0927 21:39:07.883955 (+ 2274us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.883974 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.884157 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.885233 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.885243 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.885293 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:07.885933 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.885945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.887846 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.887854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.890865 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.890883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.892746 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.892752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.892946 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.893018 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 21:39:07.893149 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 21:39:07.893356 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"mutex_wait_us":106,"spinlock_wait_cycles":2816,"thread_start_us":1722,"threads_started":1}
I20260927 21:39:07.894233 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.882149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.0.202.129:39613)
0927 21:39:07.882822 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.882836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.883999 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 21:39:07.884433 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.884442 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.884743 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.884953 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:07.886352 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.887918 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.891073 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.892669 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.892823 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.893046 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.893659 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.893846 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":280,"threads_started":1}
I20260927 21:39:07.894755 2439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.884738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.0.202.130:39433)
0927 21:39:07.885809 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.885819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.886746 (+ 927us) server_negotiation.cc:407] Connection header received
0927 21:39:07.886885 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.886893 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.887153 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.887351 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:07.887918 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.889244 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.891379 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.892891 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.893032 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.893229 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:07.894238 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0927 21:39:07.894454 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":736,"threads_started":1}
I20260927 21:39:07.895120 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.885297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39433 (local address 127.0.0.1:60724)
0927 21:39:07.886609 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.886630 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.886757 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.887354 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.887359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.887382 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:07.887705 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.887715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.889452 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.889460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.891214 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.891227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.893368 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.893375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.893612 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.893643 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:39:07.893728 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:39:07.894536 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":6,"thread_start_us":1017,"threads_started":1}
I20260927 21:39:07.895213 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.881654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:39613 (local address 127.0.0.1:43386)
0927 21:39:07.883830 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.883880 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.884157 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.885238 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.885247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.885299 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:07.886100 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.886134 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.888097 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.888104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.890865 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.890883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.893217 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.893224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:07.893427 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:07.893462 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:39:07.894502 (+ 1040us) client_negotiation.cc:240] Negotiation successful
0927 21:39:07.894661 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"mutex_wait_us":47,"spinlock_wait_cycles":3200,"thread_start_us":1911,"threads_started":1}
I20260927 21:39:07.898257 2352 tablet_service.cc:1467] Processing CreateTablet for tablet 5f13016801f94453b4dafff9bdfe7bf2 (DEFAULT_TABLE table=test-table [id=893b14e3d3ef410bbaa341027799763d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:07.898578 2213 tablet_service.cc:1467] Processing CreateTablet for tablet 5f13016801f94453b4dafff9bdfe7bf2 (DEFAULT_TABLE table=test-table [id=893b14e3d3ef410bbaa341027799763d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:07.898578 2074 tablet_service.cc:1467] Processing CreateTablet for tablet 5f13016801f94453b4dafff9bdfe7bf2 (DEFAULT_TABLE table=test-table [id=893b14e3d3ef410bbaa341027799763d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:07.899705 2352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f13016801f94453b4dafff9bdfe7bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:07.899962 2074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f13016801f94453b4dafff9bdfe7bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:07.899992 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f13016801f94453b4dafff9bdfe7bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:07.913193 2441 tablet_bootstrap.cc:492] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Bootstrap starting.
I20260927 21:39:07.916896 2441 tablet_bootstrap.cc:654] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:07.918026 2441 log.cc:824] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:07.919915 2442 tablet_bootstrap.cc:492] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Bootstrap starting.
I20260927 21:39:07.919915 2443 tablet_bootstrap.cc:492] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Bootstrap starting.
I20260927 21:39:07.921303 2441 tablet_bootstrap.cc:492] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: No bootstrap required, opened a new log
I20260927 21:39:07.921598 2441 ts_tablet_manager.cc:1397] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 21:39:07.923750 2443 tablet_bootstrap.cc:654] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:07.923748 2442 tablet_bootstrap.cc:654] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:07.925057 2443 log.cc:824] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:07.925051 2442 log.cc:824] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:07.929757 2443 tablet_bootstrap.cc:492] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: No bootstrap required, opened a new log
I20260927 21:39:07.929757 2442 tablet_bootstrap.cc:492] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: No bootstrap required, opened a new log
I20260927 21:39:07.930038 2442 ts_tablet_manager.cc:1397] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 21:39:07.930038 2443 ts_tablet_manager.cc:1397] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 21:39:07.931584 2441 raft_consensus.cc:348] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:07.932046 2441 raft_consensus.cc:374] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:07.932168 2441 raft_consensus.cc:729] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473d5fd7ad7a496fab21b586d1db400a, State: Initialized, Role: FOLLOWER
I20260927 21:39:07.932873 2441 consensus_queue.cc:260] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:07.937644 2422 heartbeater.cc:503] Master 127.0.202.190:38973 was elected leader, sending a full tablet report...
I20260927 21:39:07.938292 2441 ts_tablet_manager.cc:1428] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260927 21:39:07.940351 2442 raft_consensus.cc:348] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:07.940742 2442 raft_consensus.cc:374] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:07.940864 2442 raft_consensus.cc:729] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e095308069841bd82e664bcc38dc536, State: Initialized, Role: FOLLOWER
I20260927 21:39:07.941473 2442 consensus_queue.cc:260] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:07.942260 2443 raft_consensus.cc:348] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:07.942759 2443 raft_consensus.cc:374] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:07.942878 2443 raft_consensus.cc:729] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6475cc3fabfa456a9ebc67bf01cf9376, State: Initialized, Role: FOLLOWER
I20260927 21:39:07.943528 2443 consensus_queue.cc:260] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:07.943988 2442 ts_tablet_manager.cc:1428] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 21:39:07.945552 2283 heartbeater.cc:503] Master 127.0.202.190:38973 was elected leader, sending a full tablet report...
I20260927 21:39:07.945814 2443 ts_tablet_manager.cc:1428] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260927 21:39:07.974825 2145 tablet.cc:2369] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:07.992903 2448 raft_consensus.cc:482] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:07.993319 2448 raft_consensus.cc:504] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:07.995271 2448 leader_election.cc:288] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131:42839), 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130:39433)
I20260927 21:39:08.003386 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.995655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:47927 (local address 127.0.202.131:42839)
0927 21:39:07.995869 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.995878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.997138 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 21:39:07.997378 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.997387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.997475 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.997553 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:07.998249 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.999068 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.001294 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.002292 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.002436 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.002497 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.002951 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.003104 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:39:08.003453 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.995575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42839 (local address 127.0.202.129:47927)
0927 21:39:07.996992 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.997014 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.997191 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.997627 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.997635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.997667 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:07.998087 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.998098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.999228 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.999235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.001124 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.001136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.002528 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.002536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.002720 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.002774 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.002890 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.003059 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":4,"thread_start_us":500,"threads_started":1}
I20260927 21:39:08.004012 2439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.995906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:59047 (local address 127.0.202.130:39433)
0927 21:39:07.996155 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:07.996162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:07.997257 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 21:39:07.997472 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:07.997476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:07.997583 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:07.997696 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:07.998413 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.999374 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.001688 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.002701 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.002907 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.002982 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.003504 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.003747 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 21:39:08.004417 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:07.995802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39433 (local address 127.0.202.129:59047)
0927 21:39:07.997122 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:07.997142 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:07.997301 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:07.997735 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:07.997739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:07.997797 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:07.998200 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:07.998212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:07.999594 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:07.999598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.001476 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.001503 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.003029 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.003037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.003261 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.003301 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.003451 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.003959 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":51,"thread_start_us":570,"threads_started":1}
I20260927 21:39:08.004823 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f13016801f94453b4dafff9bdfe7bf2" candidate_uuid: "2e095308069841bd82e664bcc38dc536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473d5fd7ad7a496fab21b586d1db400a" is_pre_election: true
I20260927 21:39:08.005486 2376 raft_consensus.cc:2454] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e095308069841bd82e664bcc38dc536 in term 0.
I20260927 21:39:08.005697 2238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f13016801f94453b4dafff9bdfe7bf2" candidate_uuid: "2e095308069841bd82e664bcc38dc536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" is_pre_election: true
I20260927 21:39:08.006312 2238 raft_consensus.cc:2454] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e095308069841bd82e664bcc38dc536 in term 0.
I20260927 21:39:08.006376 2026 leader_election.cc:302] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e095308069841bd82e664bcc38dc536, 473d5fd7ad7a496fab21b586d1db400a; no voters:
I20260927 21:39:08.006914 2448 raft_consensus.cc:2781] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:08.007107 2448 raft_consensus.cc:482] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:08.007252 2448 raft_consensus.cc:3037] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.010317 2448 raft_consensus.cc:504] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
I20260927 21:39:08.011031 2448 leader_election.cc:288] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [CANDIDATE]: Term 1 election: Requested vote from peers 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131:42839), 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130:39433)
I20260927 21:39:08.011544 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f13016801f94453b4dafff9bdfe7bf2" candidate_uuid: "2e095308069841bd82e664bcc38dc536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473d5fd7ad7a496fab21b586d1db400a"
I20260927 21:39:08.011668 2238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f13016801f94453b4dafff9bdfe7bf2" candidate_uuid: "2e095308069841bd82e664bcc38dc536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6475cc3fabfa456a9ebc67bf01cf9376"
I20260927 21:39:08.011772 2376 raft_consensus.cc:3037] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.011931 2238 raft_consensus.cc:3037] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.014176 2238 raft_consensus.cc:2454] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e095308069841bd82e664bcc38dc536 in term 1.
I20260927 21:39:08.014390 2376 raft_consensus.cc:2454] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e095308069841bd82e664bcc38dc536 in term 1.
I20260927 21:39:08.014775 2024 leader_election.cc:302] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e095308069841bd82e664bcc38dc536, 6475cc3fabfa456a9ebc67bf01cf9376; no voters:
I20260927 21:39:08.015148 2448 raft_consensus.cc:2781] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:08.015683 2448 raft_consensus.cc:686] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 1 LEADER]: Becoming Leader. State: Replica: 2e095308069841bd82e664bcc38dc536, State: Running, Role: LEADER
I20260927 21:39:08.016273 2448 consensus_queue.cc:237] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } }
W20260927 21:39:08.019233 2284 tablet.cc:2369] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:08.021284 1936 catalog_manager.cc:5495] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e095308069841bd82e664bcc38dc536 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "2e095308069841bd82e664bcc38dc536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } health_report { overall_health: UNKNOWN } } }
W20260927 21:39:08.046662 2423 tablet.cc:2369] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:08.055155 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.049479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.202.129:39613)
0927 21:39:08.049740 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.049749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.049872 (+ 123us) server_negotiation.cc:407] Connection header received
0927 21:39:08.050216 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.050220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.050298 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.050447 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:08.050814 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.051533 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.053004 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.053413 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.053553 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.054049 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.054552 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.054729 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.054810 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.054890 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:39:08.064424 2238 raft_consensus.cc:1261] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Refusing update from remote peer 2e095308069841bd82e664bcc38dc536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:08.064427 2376 raft_consensus.cc:1261] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Refusing update from remote peer 2e095308069841bd82e664bcc38dc536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:08.065420 2452 consensus_queue.cc:1035] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:08.065779 2448 consensus_queue.cc:1035] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:08.080107 2456 mvcc.cc:205] Tried to move back new op lower bound from 7334072926461116416 to 7334072926279483392. Current Snapshot: MvccSnapshot[applied={T|T < 7334072926461116416}]
I20260927 21:39:08.101146 1936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:08.101529 1936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:08.105849 1936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde: Sending DeleteTablet for 3 replicas of tablet 5f13016801f94453b4dafff9bdfe7bf2
I20260927 21:39:08.107319 2352 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f13016801f94453b4dafff9bdfe7bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:08 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:46286
I20260927 21:39:08.107326 2213 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f13016801f94453b4dafff9bdfe7bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:08 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60724
I20260927 21:39:08.107321 2074 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f13016801f94453b4dafff9bdfe7bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:08 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:43386
I20260927 21:39:08.110025 1937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:08.110766 1937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:08.119540 2073 tablet_service.cc:1467] Processing CreateTablet for tablet a7b9d18e5911440690b8cc2124a63161 (DEFAULT_TABLE table=test-table [id=8c7e4fee508d4a9c86b4bf86bf19a7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:08.120954 2073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b9d18e5911440690b8cc2124a63161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:08.124483 2465 tablet_replica.cc:330] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: stopping tablet replica
I20260927 21:39:08.124869 2351 tablet_service.cc:1467] Processing CreateTablet for tablet a7b9d18e5911440690b8cc2124a63161 (DEFAULT_TABLE table=test-table [id=8c7e4fee508d4a9c86b4bf86bf19a7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:08.125046 2465 raft_consensus.cc:2229] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:08.125548 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b9d18e5911440690b8cc2124a63161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:08.125628 2465 raft_consensus.cc:2258] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:08.126915 2212 tablet_service.cc:1467] Processing CreateTablet for tablet a7b9d18e5911440690b8cc2124a63161 (DEFAULT_TABLE table=test-table [id=8c7e4fee508d4a9c86b4bf86bf19a7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:08.127576 2212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b9d18e5911440690b8cc2124a63161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:08.133570 2441 tablet_bootstrap.cc:492] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: Bootstrap starting.
I20260927 21:39:08.134327 2464 tablet_replica.cc:330] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: stopping tablet replica
I20260927 21:39:08.134879 2464 raft_consensus.cc:2229] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:08.135305 2441 tablet_bootstrap.cc:654] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:08.135510 2464 raft_consensus.cc:2258] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:08.138100 2465 ts_tablet_manager.cc:1905] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:08.140143 2443 tablet_bootstrap.cc:492] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: Bootstrap starting.
I20260927 21:39:08.142401 2443 tablet_bootstrap.cc:654] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:08.143584 2442 tablet_bootstrap.cc:492] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: Bootstrap starting.
I20260927 21:39:08.144662 2464 ts_tablet_manager.cc:1905] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:08.145572 2442 tablet_bootstrap.cc:654] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:08.146189 2441 tablet_bootstrap.cc:492] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: No bootstrap required, opened a new log
I20260927 21:39:08.146394 2441 ts_tablet_manager.cc:1397] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260927 21:39:08.146992 2466 tablet_replica.cc:330] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: stopping tablet replica
I20260927 21:39:08.147380 2441 raft_consensus.cc:348] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.147675 2441 raft_consensus.cc:374] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:08.147806 2441 raft_consensus.cc:729] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473d5fd7ad7a496fab21b586d1db400a, State: Initialized, Role: FOLLOWER
I20260927 21:39:08.148180 2466 raft_consensus.cc:2229] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:08.148164 2441 consensus_queue.cc:260] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.151238 2441 ts_tablet_manager.cc:1428] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 21:39:08.151996 2466 raft_consensus.cc:2258] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:08.155506 2464 ts_tablet_manager.cc:1918] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:08.155758 2464 log.cc:1196] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/5f13016801f94453b4dafff9bdfe7bf2
I20260927 21:39:08.156467 2464 ts_tablet_manager.cc:1939] T 5f13016801f94453b4dafff9bdfe7bf2 P 6475cc3fabfa456a9ebc67bf01cf9376: Deleting consensus metadata
I20260927 21:39:08.158108 1921 catalog_manager.cc:4841] TS 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130:39433): tablet 5f13016801f94453b4dafff9bdfe7bf2 (table test-table [id=893b14e3d3ef410bbaa341027799763d]) successfully deleted
I20260927 21:39:08.158751 2443 tablet_bootstrap.cc:492] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: No bootstrap required, opened a new log
I20260927 21:39:08.158900 2443 ts_tablet_manager.cc:1397] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260927 21:39:08.159765 2465 ts_tablet_manager.cc:1918] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:08.159701 2443 raft_consensus.cc:348] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.159901 2443 raft_consensus.cc:374] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:08.159938 2465 log.cc:1196] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/5f13016801f94453b4dafff9bdfe7bf2
I20260927 21:39:08.160002 2443 raft_consensus.cc:729] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6475cc3fabfa456a9ebc67bf01cf9376, State: Initialized, Role: FOLLOWER
I20260927 21:39:08.160154 2466 ts_tablet_manager.cc:1905] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:08.160390 2443 consensus_queue.cc:260] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.160548 2465 ts_tablet_manager.cc:1939] T 5f13016801f94453b4dafff9bdfe7bf2 P 473d5fd7ad7a496fab21b586d1db400a: Deleting consensus metadata
I20260927 21:39:08.161813 2443 ts_tablet_manager.cc:1428] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:39:08.161847 2442 tablet_bootstrap.cc:492] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: No bootstrap required, opened a new log
I20260927 21:39:08.162110 2442 ts_tablet_manager.cc:1397] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20260927 21:39:08.163195 2442 raft_consensus.cc:348] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.163497 2442 raft_consensus.cc:374] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:08.163643 2442 raft_consensus.cc:729] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e095308069841bd82e664bcc38dc536, State: Initialized, Role: FOLLOWER
I20260927 21:39:08.164021 2442 consensus_queue.cc:260] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.164815 2442 ts_tablet_manager.cc:1428] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:39:08.164824 1923 catalog_manager.cc:4841] TS 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131:42839): tablet 5f13016801f94453b4dafff9bdfe7bf2 (table test-table [id=893b14e3d3ef410bbaa341027799763d]) successfully deleted
I20260927 21:39:08.171139 2466 ts_tablet_manager.cc:1918] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:08.171332 2466 log.cc:1196] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/5f13016801f94453b4dafff9bdfe7bf2
I20260927 21:39:08.171911 2466 ts_tablet_manager.cc:1939] T 5f13016801f94453b4dafff9bdfe7bf2 P 2e095308069841bd82e664bcc38dc536: Deleting consensus metadata
I20260927 21:39:08.173766 1921 catalog_manager.cc:4841] TS 2e095308069841bd82e664bcc38dc536 (127.0.202.129:39613): tablet 5f13016801f94453b4dafff9bdfe7bf2 (table test-table [id=893b14e3d3ef410bbaa341027799763d]) successfully deleted
I20260927 21:39:08.312191 2449 raft_consensus.cc:482] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:08.312507 2449 raft_consensus.cc:504] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.314085 2449 leader_election.cc:288] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e095308069841bd82e664bcc38dc536 (127.0.202.129:39613), 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131:42839)
I20260927 21:39:08.322968 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.314586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:54683 (local address 127.0.202.129:39613)
0927 21:39:08.315007 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.315017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.316185 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 21:39:08.316514 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.316522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.316604 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.316653 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:08.317567 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.318313 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.321187 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.321968 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.322065 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.322117 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.322529 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.322667 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260927 21:39:08.323086 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.314605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:39613 (local address 127.0.202.130:54683)
0927 21:39:08.316057 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.316080 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.316337 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.316832 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.316835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.316966 (+ 131us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:08.317399 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.317407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.318456 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.318462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.321005 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.321021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.322261 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.322265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.322377 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.322398 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.322505 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.322634 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"mutex_wait_us":56,"spinlock_wait_cycles":131456,"thread_start_us":1374,"threads_started":1}
I20260927 21:39:08.323602 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.315217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:53301 (local address 127.0.202.131:42839)
0927 21:39:08.315435 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.315440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.316444 (+ 1004us) server_negotiation.cc:407] Connection header received
0927 21:39:08.316582 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.316590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.316679 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.316776 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:08.317563 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.318621 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.321199 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.322419 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.322568 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.322642 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.323225 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.323363 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:39:08.323971 2099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b9d18e5911440690b8cc2124a63161" candidate_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e095308069841bd82e664bcc38dc536" is_pre_election: true
I20260927 21:39:08.324398 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.314728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42839 (local address 127.0.202.130:53301)
0927 21:39:08.316281 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.316326 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.316458 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.316824 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.316829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.316853 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:08.317327 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.317344 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.318756 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.318761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.321049 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.321061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.322802 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.322809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.322994 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.323050 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.323843 (+ 793us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.323979 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"mutex_wait_us":46,"thread_start_us":893,"threads_started":1}
I20260927 21:39:08.324465 2099 raft_consensus.cc:2454] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6475cc3fabfa456a9ebc67bf01cf9376 in term 0.
I20260927 21:39:08.325107 2164 leader_election.cc:302] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e095308069841bd82e664bcc38dc536, 6475cc3fabfa456a9ebc67bf01cf9376; no voters:
I20260927 21:39:08.325162 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b9d18e5911440690b8cc2124a63161" candidate_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473d5fd7ad7a496fab21b586d1db400a" is_pre_election: true
I20260927 21:39:08.325387 2376 raft_consensus.cc:2454] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6475cc3fabfa456a9ebc67bf01cf9376 in term 0.
I20260927 21:39:08.325507 2449 raft_consensus.cc:2781] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:08.325614 2449 raft_consensus.cc:482] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:08.325699 2449 raft_consensus.cc:3037] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.328927 2449 raft_consensus.cc:504] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.329638 2449 leader_election.cc:288] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [CANDIDATE]: Term 1 election: Requested vote from peers 2e095308069841bd82e664bcc38dc536 (127.0.202.129:39613), 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131:42839)
I20260927 21:39:08.330137 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b9d18e5911440690b8cc2124a63161" candidate_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473d5fd7ad7a496fab21b586d1db400a"
I20260927 21:39:08.330143 2099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b9d18e5911440690b8cc2124a63161" candidate_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e095308069841bd82e664bcc38dc536"
I20260927 21:39:08.330340 2376 raft_consensus.cc:3037] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.330377 2099 raft_consensus.cc:3037] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.333302 2376 raft_consensus.cc:2454] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6475cc3fabfa456a9ebc67bf01cf9376 in term 1.
I20260927 21:39:08.333688 2099 raft_consensus.cc:2454] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6475cc3fabfa456a9ebc67bf01cf9376 in term 1.
I20260927 21:39:08.334174 2166 leader_election.cc:302] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 473d5fd7ad7a496fab21b586d1db400a, 6475cc3fabfa456a9ebc67bf01cf9376; no voters:
I20260927 21:39:08.334517 2449 raft_consensus.cc:2781] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:08.334959 2449 raft_consensus.cc:686] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 LEADER]: Becoming Leader. State: Replica: 6475cc3fabfa456a9ebc67bf01cf9376, State: Running, Role: LEADER
I20260927 21:39:08.335351 2449 consensus_queue.cc:237] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.339236 1936 catalog_manager.cc:5495] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130). New cstate: current_term: 1 leader_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:39:08 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:08.406610 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.391089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:38973 (local address 127.0.0.1:59892)
0927 21:39:08.391857 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.391883 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.392055 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.392644 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.392649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.393558 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:08.393851 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.393860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.395051 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.395058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.396214 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.396227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.396949 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.396954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.397130 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.397858 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:08.397875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:08.400047 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:08.403564 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:08.403579 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:08.403589 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:08.404037 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:08.404503 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:08.404509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:08.404513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:08.404720 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:08.405270 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:08.405278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:08.405510 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.405719 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.406062 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":380,"threads_started":1}
I20260927 21:39:08.406646 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.391525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.0.202.190:38973)
0927 21:39:08.392186 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.392193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.392216 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:39:08.392273 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.392276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.392419 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.392636 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:08.394091 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.394907 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.396410 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.396802 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.396934 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.400244 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:08.400249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:08.400252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:08.400280 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:08.403351 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.404179 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.404184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.404190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.404387 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.404828 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.404837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.404843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.405127 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:08.405278 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.405862 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.406210 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":517,"thread_start_us":374,"threads_started":1}
I20260927 21:39:08.413307 1936 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:59892:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:39:08.414371 1936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:08.421816 2074 tablet_service.cc:1467] Processing CreateTablet for tablet 6f630ae9734445329b4f7fd01d881618 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=112c9f9d0a0c4e80b9aba2b4d0e31d44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:39:08.422183 2351 tablet_service.cc:1467] Processing CreateTablet for tablet 6f630ae9734445329b4f7fd01d881618 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=112c9f9d0a0c4e80b9aba2b4d0e31d44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:39:08.422401 2074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f630ae9734445329b4f7fd01d881618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:08.422729 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f630ae9734445329b4f7fd01d881618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:08.427644 2212 tablet_service.cc:1467] Processing CreateTablet for tablet 6f630ae9734445329b4f7fd01d881618 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=112c9f9d0a0c4e80b9aba2b4d0e31d44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:39:08.428234 2212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f630ae9734445329b4f7fd01d881618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:08.430994 2441 tablet_bootstrap.cc:492] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: Bootstrap starting.
I20260927 21:39:08.431423 2442 tablet_bootstrap.cc:492] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: Bootstrap starting.
I20260927 21:39:08.433463 2441 tablet_bootstrap.cc:654] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:08.433823 2442 tablet_bootstrap.cc:654] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:08.441171 2443 tablet_bootstrap.cc:492] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: Bootstrap starting.
I20260927 21:39:08.443248 2443 tablet_bootstrap.cc:654] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:08.443699 2442 tablet_bootstrap.cc:492] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: No bootstrap required, opened a new log
I20260927 21:39:08.443894 2442 ts_tablet_manager.cc:1397] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260927 21:39:08.445379 2442 raft_consensus.cc:348] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.445740 2442 raft_consensus.cc:374] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:08.445843 2442 raft_consensus.cc:729] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e095308069841bd82e664bcc38dc536, State: Initialized, Role: FOLLOWER
I20260927 21:39:08.446239 2442 consensus_queue.cc:260] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.446832 2452 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.447232 2452 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:08.447923 2442 ts_tablet_manager.cc:1428] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:39:08.450500 2441 tablet_bootstrap.cc:492] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: No bootstrap required, opened a new log
I20260927 21:39:08.450757 2441 ts_tablet_manager.cc:1397] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20260927 21:39:08.451785 2441 raft_consensus.cc:348] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.452127 2441 raft_consensus.cc:374] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:08.452227 2441 raft_consensus.cc:729] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473d5fd7ad7a496fab21b586d1db400a, State: Initialized, Role: FOLLOWER
I20260927 21:39:08.452575 2441 consensus_queue.cc:260] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.453235 2441 ts_tablet_manager.cc:1428] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:39:08.453140 2447 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.453488 2447 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:08.458292 2443 tablet_bootstrap.cc:492] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: No bootstrap required, opened a new log
I20260927 21:39:08.458484 2443 ts_tablet_manager.cc:1397] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.014s
I20260927 21:39:08.459143 2443 raft_consensus.cc:348] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.459357 2443 raft_consensus.cc:374] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:08.459434 2443 raft_consensus.cc:729] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6475cc3fabfa456a9ebc67bf01cf9376, State: Initialized, Role: FOLLOWER
I20260927 21:39:08.459681 2443 consensus_queue.cc:260] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.460243 2443 ts_tablet_manager.cc:1428] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 21:39:08.460141 2449 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.460480 2449 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:08.600602 2447 raft_consensus.cc:482] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:08.600883 2447 raft_consensus.cc:504] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.602437 2447 leader_election.cc:288] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e095308069841bd82e664bcc38dc536 (127.0.202.129:39613), 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130:39433)
I20260927 21:39:08.610476 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.603097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:52849 (local address 127.0.202.129:39613)
0927 21:39:08.603341 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.603349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.603583 (+ 234us) server_negotiation.cc:407] Connection header received
0927 21:39:08.603877 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.603888 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.603984 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.604123 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:08.604700 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.605782 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.607984 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.609011 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.609649 (+ 638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.609733 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.610041 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.610195 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 21:39:08.610615 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.602896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:39613 (local address 127.0.202.131:52849)
0927 21:39:08.603439 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.603467 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.603686 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.604155 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.604161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.604191 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:08.604548 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.604576 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.605928 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.605934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.607821 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.607833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.609651 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.609658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.609862 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.609888 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.610000 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.610172 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":1,"thread_start_us":316,"threads_started":1}
I20260927 21:39:08.611127 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.603666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39433 (local address 127.0.202.131:44005)
0927 21:39:08.604399 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.604424 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.604549 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.604966 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.604971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.604998 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:08.605333 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.605345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.606896 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.606900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.608707 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.608719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.610182 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.610189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.610387 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.610413 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.610529 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.610705 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":41,"thread_start_us":231,"threads_started":1}
I20260927 21:39:08.611317 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.603814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:44005 (local address 127.0.202.130:39433)
0927 21:39:08.604681 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.604688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.604707 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:39:08.604775 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.604779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.604836 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.604919 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:08.605535 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.606744 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.608866 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.609958 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.610160 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.610240 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.610723 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.610887 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":614,"threads_started":1}
I20260927 21:39:08.611320 2099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f630ae9734445329b4f7fd01d881618" candidate_uuid: "473d5fd7ad7a496fab21b586d1db400a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e095308069841bd82e664bcc38dc536" is_pre_election: true
I20260927 21:39:08.611639 2099 raft_consensus.cc:2454] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 473d5fd7ad7a496fab21b586d1db400a in term 0.
I20260927 21:39:08.611892 2238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f630ae9734445329b4f7fd01d881618" candidate_uuid: "473d5fd7ad7a496fab21b586d1db400a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" is_pre_election: true
I20260927 21:39:08.612135 2238 raft_consensus.cc:2454] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 473d5fd7ad7a496fab21b586d1db400a in term 0.
I20260927 21:39:08.612187 2303 leader_election.cc:302] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e095308069841bd82e664bcc38dc536, 473d5fd7ad7a496fab21b586d1db400a; no voters:
I20260927 21:39:08.612735 2447 raft_consensus.cc:2781] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:08.612844 2447 raft_consensus.cc:482] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:08.613076 2447 raft_consensus.cc:3037] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.615490 2447 raft_consensus.cc:504] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.616077 2447 leader_election.cc:288] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [CANDIDATE]: Term 1 election: Requested vote from peers 2e095308069841bd82e664bcc38dc536 (127.0.202.129:39613), 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130:39433)
I20260927 21:39:08.616739 2238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f630ae9734445329b4f7fd01d881618" candidate_uuid: "473d5fd7ad7a496fab21b586d1db400a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6475cc3fabfa456a9ebc67bf01cf9376"
I20260927 21:39:08.616765 2099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f630ae9734445329b4f7fd01d881618" candidate_uuid: "473d5fd7ad7a496fab21b586d1db400a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e095308069841bd82e664bcc38dc536"
I20260927 21:39:08.616952 2238 raft_consensus.cc:3037] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.616990 2099 raft_consensus.cc:3037] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:08.619053 2238 raft_consensus.cc:2454] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 473d5fd7ad7a496fab21b586d1db400a in term 1.
I20260927 21:39:08.619056 2099 raft_consensus.cc:2454] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 473d5fd7ad7a496fab21b586d1db400a in term 1.
I20260927 21:39:08.619465 2303 leader_election.cc:302] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 473d5fd7ad7a496fab21b586d1db400a, 6475cc3fabfa456a9ebc67bf01cf9376; no voters:
I20260927 21:39:08.619839 2447 raft_consensus.cc:2781] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:08.620327 2447 raft_consensus.cc:686] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [term 1 LEADER]: Becoming Leader. State: Replica: 473d5fd7ad7a496fab21b586d1db400a, State: Running, Role: LEADER
I20260927 21:39:08.620785 2447 consensus_queue.cc:237] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } }
I20260927 21:39:08.622784 2487 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: TxnStatusTablet state changed. Reason: New leader 473d5fd7ad7a496fab21b586d1db400a. Latest consensus state: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.623157 2487 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:39:08.623951 2489 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:39:08.624197 2489 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:39:08.624886 1936 catalog_manager.cc:5495] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a reported cstate change: term changed from 0 to 1, leader changed from <none> to 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131). New cstate: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } health_report { overall_health: HEALTHY } } }
I20260927 21:39:08.687668 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.680426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42839 (local address 127.0.0.1:46300)
0927 21:39:08.680662 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.680683 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.680831 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.681284 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.681290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.681305 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:08.681504 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.681514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.682547 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.682562 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.684745 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.684760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.685780 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.685789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.685985 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.686042 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.687079 (+ 1037us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.687100 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.687220 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.687372 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 21:39:08.687673 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.680475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.202.131:42839)
0927 21:39:08.680707 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.680715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.680732 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:39:08.681031 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.681034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.681109 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.681239 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:08.681656 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.682422 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.684896 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.685569 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.685712 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.686279 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.686963 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.687118 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.687239 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.687324 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 21:39:08.691871 2238 raft_consensus.cc:1261] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Refusing update from remote peer 473d5fd7ad7a496fab21b586d1db400a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:08.692276 2099 raft_consensus.cc:1261] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Refusing update from remote peer 473d5fd7ad7a496fab21b586d1db400a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:08.692525 2447 consensus_queue.cc:1035] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:08.692970 2487 consensus_queue.cc:1035] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:08.696234 2452 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: TxnStatusTablet state changed. Reason: New leader 473d5fd7ad7a496fab21b586d1db400a. Latest consensus state: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.696565 2452 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:08.699270 2447 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.699585 2447 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:39:08.712096 2452 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.712462 2452 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 2e095308069841bd82e664bcc38dc536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:08.717286 2449 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: TxnStatusTablet state changed. Reason: New leader 473d5fd7ad7a496fab21b586d1db400a. Latest consensus state: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.717763 2449 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:08.717639 2487 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.717952 2487 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 473d5fd7ad7a496fab21b586d1db400a: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:39:08.732590 2493 mvcc.cc:205] Tried to move back new op lower bound from 7334072929035149312 to 7334072928754520064. Current Snapshot: MvccSnapshot[applied={T|T < 7334072929035149312}]
I20260927 21:39:08.741451 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.714478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.0.202.130:39433)
0927 21:39:08.714709 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.714717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.729918 (+ 15201us) server_negotiation.cc:407] Connection header received
0927 21:39:08.730619 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.730628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.730829 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.731008 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:08.731900 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.733626 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.736956 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.738022 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.738185 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.739252 (+ 1067us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.740323 (+ 1071us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.740492 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.740850 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.740988 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:39:08.745141 2449 tablet_replica.cc:439] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "473d5fd7ad7a496fab21b586d1db400a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 } } peers { permanent_uuid: "6475cc3fabfa456a9ebc67bf01cf9376" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39433 } } peers { permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 } } }
I20260927 21:39:08.745481 2449 tablet_replica.cc:442] T 6f630ae9734445329b4f7fd01d881618 P 6475cc3fabfa456a9ebc67bf01cf9376: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:08.767521 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.754915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:39537 (local address 127.0.202.131:42839)
0927 21:39:08.755145 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.755156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.760261 (+ 5105us) server_negotiation.cc:407] Connection header received
0927 21:39:08.760535 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.760543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.760628 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.760710 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:08.761401 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.762567 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.765100 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.765948 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.766114 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.766636 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.766781 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.766881 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.767042 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.767140 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:39:08.767545 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.754908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42839 (local address 127.0.202.130:39537)
0927 21:39:08.760128 (+ 5220us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.760150 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.760348 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.760863 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.760868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.760880 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:08.761221 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.761234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.762733 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.762738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.764928 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.764944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.766179 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.766185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.766352 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.766427 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.766899 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.766911 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.766998 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.767159 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5039,"thread_start_us":4901,"threads_started":1}
I20260927 21:39:08.778367 2376 raft_consensus.cc:1261] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Refusing update from remote peer 6475cc3fabfa456a9ebc67bf01cf9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:08.778484 2099 raft_consensus.cc:1261] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Refusing update from remote peer 6475cc3fabfa456a9ebc67bf01cf9376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:08.779186 2472 consensus_queue.cc:1035] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e095308069841bd82e664bcc38dc536" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:08.780382 2472 consensus_queue.cc:1035] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "473d5fd7ad7a496fab21b586d1db400a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:08.833909 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.826479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:33621 (local address 127.0.202.130:39433)
0927 21:39:08.826690 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.826696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.827444 (+ 748us) server_negotiation.cc:407] Connection header received
0927 21:39:08.827601 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.827606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.827689 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.827780 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:08.828537 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.829598 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.831775 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.832468 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.832597 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.833114 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.833264 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.833352 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.833519 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.833608 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:39:08.834358 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.826181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39433 (local address 127.0.202.131:33621)
0927 21:39:08.827316 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:08.827337 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:08.827481 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:08.827824 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:08.827830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:08.827885 (+ 55us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:08.828344 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.828357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.829754 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.829759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.831620 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:08.831635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.832663 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.832670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:08.832856 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.832929 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:08.833374 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:08.833385 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:39:08.833480 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:39:08.833626 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":690,"threads_started":1}
I20260927 21:39:08.890412 1937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:08.890694 1937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:08.894171 1937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d82af54d1f4916afcac3811f064fde: Sending DeleteTablet for 3 replicas of tablet a7b9d18e5911440690b8cc2124a63161
I20260927 21:39:08.895191 2351 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b9d18e5911440690b8cc2124a63161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:08 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:46286
I20260927 21:39:08.895128 2074 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b9d18e5911440690b8cc2124a63161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:08 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:43386
I20260927 21:39:08.897248 2212 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b9d18e5911440690b8cc2124a63161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:08 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60724
I20260927 21:39:08.900789 2513 tablet_replica.cc:330] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: stopping tablet replica
I20260927 21:39:08.900980 2515 tablet_replica.cc:330] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: stopping tablet replica
I20260927 21:39:08.901024 2513 raft_consensus.cc:2229] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:08.901314 2513 raft_consensus.cc:2258] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:08.901290 2515 raft_consensus.cc:2229] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:08.901916 2515 raft_consensus.cc:2258] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:08.903129 2515 ts_tablet_manager.cc:1905] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:08.904111 2513 ts_tablet_manager.cc:1905] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:08.904755 2512 tablet_replica.cc:330] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: stopping tablet replica
I20260927 21:39:08.905488 2512 raft_consensus.cc:2229] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:08.907115 2512 raft_consensus.cc:2258] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:08.908257 2512 ts_tablet_manager.cc:1905] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:08.913131 2513 ts_tablet_manager.cc:1918] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:08.913280 2513 log.cc:1196] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/a7b9d18e5911440690b8cc2124a63161
I20260927 21:39:08.913730 2513 ts_tablet_manager.cc:1939] T a7b9d18e5911440690b8cc2124a63161 P 2e095308069841bd82e664bcc38dc536: Deleting consensus metadata
I20260927 21:39:08.914832 1921 catalog_manager.cc:4841] TS 2e095308069841bd82e664bcc38dc536 (127.0.202.129:39613): tablet a7b9d18e5911440690b8cc2124a63161 (table test-table [id=8c7e4fee508d4a9c86b4bf86bf19a7e3]) successfully deleted
I20260927 21:39:08.917591 2515 ts_tablet_manager.cc:1918] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:08.921911 2512 ts_tablet_manager.cc:1918] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:08.922312 2515 log.cc:1196] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/a7b9d18e5911440690b8cc2124a63161
I20260927 21:39:08.922961 2515 ts_tablet_manager.cc:1939] T a7b9d18e5911440690b8cc2124a63161 P 6475cc3fabfa456a9ebc67bf01cf9376: Deleting consensus metadata
I20260927 21:39:08.924196 1921 catalog_manager.cc:4841] TS 6475cc3fabfa456a9ebc67bf01cf9376 (127.0.202.130:39433): tablet a7b9d18e5911440690b8cc2124a63161 (table test-table [id=8c7e4fee508d4a9c86b4bf86bf19a7e3]) successfully deleted
I20260927 21:39:08.924649 2512 log.cc:1196] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/a7b9d18e5911440690b8cc2124a63161
I20260927 21:39:08.925190 2512 ts_tablet_manager.cc:1939] T a7b9d18e5911440690b8cc2124a63161 P 473d5fd7ad7a496fab21b586d1db400a: Deleting consensus metadata
I20260927 21:39:08.926457 1923 catalog_manager.cc:4841] TS 473d5fd7ad7a496fab21b586d1db400a (127.0.202.131:42839): tablet a7b9d18e5911440690b8cc2124a63161 (table test-table [id=8c7e4fee508d4a9c86b4bf86bf19a7e3]) successfully deleted
Sep 27 21:39:08 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:08.937968 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.910821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.0.202.129:39613)
0927 21:39:08.911041 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.911051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.918815 (+ 7764us) server_negotiation.cc:407] Connection header received
0927 21:39:08.919098 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.919104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.919187 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.919265 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:08.921350 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.922364 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.924153 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.924811 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.924905 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.930720 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:08.930738 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:08.930758 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:08.930804 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:08.934752 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.935667 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.935679 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.935682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.935842 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.936267 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.936272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.936275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.936746 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:08.937051 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.937355 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.937598 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 21:39:08.938910 2139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43404
I20260927 21:39:08.971323 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.945778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.202.190:38973)
0927 21:39:08.953986 (+ 8208us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.953993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.954015 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:39:08.954079 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.954087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.954171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.954285 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:08.956105 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.957331 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.959180 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.959891 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.960028 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.963827 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:08.963835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:08.963841 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:08.963894 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:08.967873 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.968848 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.968855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.968859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.969037 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.969488 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.969493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.969497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.969894 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:08.970184 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.970621 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.970886 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8066,"thread_start_us":7521,"threads_started":1}
W20260927 21:39:08.972026 1937 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59896
I20260927 21:39:08.975221 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2008
I20260927 21:39:08.994455 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:08.977461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.202.129:39613)
0927 21:39:08.977694 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:08.977701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:08.977722 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:39:08.977813 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:08.977820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:08.977899 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:08.978001 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:08.979710 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.980869 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.982689 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:08.983361 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:08.983492 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:08.987436 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:08.987444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:08.987451 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:08.987500 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:08.991253 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.992214 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.992220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.992224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.992423 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:08.992861 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:08.992866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:08.992870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:08.993211 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:08.993451 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:08.993840 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 21:39:08.994068 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260927 21:39:08.995002 2139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43414
W20260927 21:39:08.995684 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:08.995898 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 2008
I20260927 21:39:09.027282 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2148
Sep 27 21:39:09 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:39:09.048062 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:09.028139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.0.202.130:39433)
0927 21:39:09.028453 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:09.028460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:09.028514 (+ 54us) server_negotiation.cc:407] Connection header received
0927 21:39:09.028731 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:09.028737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:09.028818 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:09.028943 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:09.030432 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:09.031627 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:09.033606 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:09.034321 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:09.034453 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:09.040012 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:09.040031 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:09.040053 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:09.040109 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:09.044411 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:09.045379 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:09.045402 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:09.045406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:09.045589 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:09.046042 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:09.046048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:09.046052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:09.046614 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:09.046933 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:09.047269 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 21:39:09.047503 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
W20260927 21:39:09.048858 2278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60738
W20260927 21:39:09.049665 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2148: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:09.049863 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 2148
I20260927 21:39:09.065430 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2287
Sep 27 21:39:09 dist-test-slave-25dq krb5kdc[1892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:39:09.081652 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:09.066232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.0.202.131:42839)
0927 21:39:09.066414 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:09.066420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:09.066533 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:39:09.066751 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:09.066763 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:09.066827 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:09.066942 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:09.068150 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:09.068994 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:09.070357 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:09.070818 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:09.070988 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:09.075469 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:09.075487 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:09.075503 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:09.075551 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:09.078683 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:09.079366 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:09.079384 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:09.079387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:09.079523 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:09.079904 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:09.079909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:09.079912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:09.080399 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:09.080739 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:09.080996 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 21:39:09.081243 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 21:39:09.082397 2417 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48934
W20260927 21:39:09.083112 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2287: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:09.083272 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 2287
I20260927 21:39:09.104417 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 1905
I20260927 21:39:09.105698 1977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:09.221698 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 1905
2026-09-27T21:39:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4937 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:09 dist-test-slave-25dq krb5kdc[2524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:09 dist-test-slave-25dq krb5kdc[2524](info): set up 2 sockets
Sep 27 21:39:09 dist-test-slave-25dq krb5kdc[2524](info): commencing operation
krb5kdc: starting...
W20260927 21:39:10.328590 810 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:10 dist-test-slave-25dq krb5kdc[2524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:10.525583 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40125
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:10.859171 2537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:10.859474 2537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:10.859571 2537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:10.866953 2537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:10.867110 2537 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:39:10.867202 2537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:10.867311 2537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:10.867403 2537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:10.867487 2537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:10.878152 2537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40125
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:10.879575 2537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:10.881073 2537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:10.893560 2542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:10.893749 2543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:10.894109 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:10.894941 2537 server_base.cc:1034] running on GCE node
I20260927 21:39:10.895928 2537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:10.897991 2537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:10.899354 2537 hybrid_clock.cc:648] HybridClock initialized: now 1790545150899293 us; error 61 us; skew 500 ppm
I20260927 21:39:10.899835 2537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:10 dist-test-slave-25dq krb5kdc[2524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:10.903497 2537 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:10.905249 2537 webserver.cc:460] Webserver started at http://127.0.202.190:44173/ using document root <none> and password file <none>
I20260927 21:39:10.905994 2537 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:10.906137 2537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:10.906479 2537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:10.909631 2537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "65ccad99ce80405c9ebb576c5ee231e9"
format_stamp: "Formatted at 2026-09-27 21:39:10 on dist-test-slave-25dq"
server_key: "92fb119b219fc0d7705fdc0838694c42"
server_key_iv: "4016edced5f1f814cf65169324bcdd15"
server_key_version: "encryptionkey@0"
I20260927 21:39:10.910395 2537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "65ccad99ce80405c9ebb576c5ee231e9"
format_stamp: "Formatted at 2026-09-27 21:39:10 on dist-test-slave-25dq"
server_key: "92fb119b219fc0d7705fdc0838694c42"
server_key_iv: "4016edced5f1f814cf65169324bcdd15"
server_key_version: "encryptionkey@0"
I20260927 21:39:10.916004 2537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:39:10.919756 2552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:10.921157 2537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:10.921348 2537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "65ccad99ce80405c9ebb576c5ee231e9"
format_stamp: "Formatted at 2026-09-27 21:39:10 on dist-test-slave-25dq"
server_key: "92fb119b219fc0d7705fdc0838694c42"
server_key_iv: "4016edced5f1f814cf65169324bcdd15"
server_key_version: "encryptionkey@0"
I20260927 21:39:10.921568 2537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:10.938144 2537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:10.942471 2537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:10.942750 2537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:10.965790 2537 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40125
I20260927 21:39:10.965811 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40125 every 8 connection(s)
I20260927 21:39:10.967156 2537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:10.970878 2614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:10.976076 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 2537
I20260927 21:39:10.976316 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:10.977077 810 external_mini_cluster.cc:1428] Setting key b8d13bb10bb5eafd5a75f62212436668
I20260927 21:39:10.983127 2614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Bootstrap starting.
I20260927 21:39:10.986630 2614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:10.987752 2614 log.cc:824] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:10.990957 2614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: No bootstrap required, opened a new log
Sep 27 21:39:10 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:10.999917 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:10.979359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.0.202.190:40125)
0927 21:39:10.980441 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:10.980457 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:10.980502 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:39:10.982251 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:10.982278 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:10.982687 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:10.983149 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:10.984646 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:10.985933 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:10.987267 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:10.988036 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:10.988234 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:10.992464 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:10.992472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:10.992488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:10.992528 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:10.995656 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:10.996424 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:10.996440 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:10.996451 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:10.996625 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:10.997031 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:10.997034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:10.997036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:10.997578 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:10.997937 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:10.998382 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 21:39:10.998829 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":362,"threads_started":1}
I20260927 21:39:10.999765 2614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccad99ce80405c9ebb576c5ee231e9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40125 } }
I20260927 21:39:11.000253 2614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:11.000428 2614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ccad99ce80405c9ebb576c5ee231e9, State: Initialized, Role: FOLLOWER
I20260927 21:39:11.000979 2614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccad99ce80405c9ebb576c5ee231e9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40125 } }
I20260927 21:39:11.001233 2614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:11.001338 2614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:11.001540 2614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:11.004146 2614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccad99ce80405c9ebb576c5ee231e9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40125 } }
I20260927 21:39:11.004731 2614 leader_election.cc:302] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ccad99ce80405c9ebb576c5ee231e9; no voters:
I20260927 21:39:11.005373 2614 leader_election.cc:288] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:11.005518 2619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:11.006269 2619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [term 1 LEADER]: Becoming Leader. State: Replica: 65ccad99ce80405c9ebb576c5ee231e9, State: Running, Role: LEADER
I20260927 21:39:11.006876 2619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccad99ce80405c9ebb576c5ee231e9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40125 } }
I20260927 21:39:11.007822 2614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:11.010120 2621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ccad99ce80405c9ebb576c5ee231e9. Latest consensus state: current_term: 1 leader_uuid: "65ccad99ce80405c9ebb576c5ee231e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccad99ce80405c9ebb576c5ee231e9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40125 } } }
I20260927 21:39:11.010741 2621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:11.011554 2620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ccad99ce80405c9ebb576c5ee231e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccad99ce80405c9ebb576c5ee231e9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40125 } } }
I20260927 21:39:11.011930 2620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:11.012609 2625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:11.017881 2625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:11.028716 2625 catalog_manager.cc:1347] Generated new cluster ID: dac7499fe6a1484ab138096b0f3eb78a
I20260927 21:39:11.028880 2625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:11.081720 2625 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:11.083664 2625 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:11.101538 2625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Generated new TSK 0
I20260927 21:39:11.102464 2625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:11.160565 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40125
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:11.493382 2640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:11.493731 2640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:11.493908 2640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:11.501590 2640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:11.501751 2640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:11.501900 2640 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:39:11.501986 2640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:39:11.502120 2640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:11.512499 2640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:40125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:11.514067 2640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:11.515673 2640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:11.529883 2645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:11.529902 2646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:11.529815 2648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:11.530655 2640 server_base.cc:1034] running on GCE node
I20260927 21:39:11.531329 2640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:11.532472 2640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:11.533715 2640 hybrid_clock.cc:648] HybridClock initialized: now 1790545151533637 us; error 74 us; skew 500 ppm
I20260927 21:39:11.534094 2640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:11 dist-test-slave-25dq krb5kdc[2524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:11.537339 2640 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:11.538941 2640 webserver.cc:460] Webserver started at http://127.0.202.129:35611/ using document root <none> and password file <none>
I20260927 21:39:11.539601 2640 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:11.539698 2640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:11.539937 2640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:11.542995 2640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "03525bbaa2c04165a87dba22117d5868"
format_stamp: "Formatted at 2026-09-27 21:39:11 on dist-test-slave-25dq"
server_key: "928565eefd4699be43a5fe288828e22d"
server_key_iv: "52be6eb2a919dfd930109b41c51cfca2"
server_key_version: "encryptionkey@0"
I20260927 21:39:11.543754 2640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "03525bbaa2c04165a87dba22117d5868"
format_stamp: "Formatted at 2026-09-27 21:39:11 on dist-test-slave-25dq"
server_key: "928565eefd4699be43a5fe288828e22d"
server_key_iv: "52be6eb2a919dfd930109b41c51cfca2"
server_key_version: "encryptionkey@0"
I20260927 21:39:11.549165 2640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:39:11.552775 2655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:11.554196 2640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:39:11.554358 2640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "03525bbaa2c04165a87dba22117d5868"
format_stamp: "Formatted at 2026-09-27 21:39:11 on dist-test-slave-25dq"
server_key: "928565eefd4699be43a5fe288828e22d"
server_key_iv: "52be6eb2a919dfd930109b41c51cfca2"
server_key_version: "encryptionkey@0"
I20260927 21:39:11.554560 2640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:11.569816 2640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:11.574501 2640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:11.574803 2640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:11.577697 2640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:11.577837 2640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:11.577952 2640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:11.578038 2640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:39:11 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:11.603718 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:11.578630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:44649 (local address 127.0.202.190:40125)
0927 21:39:11.579199 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:11.579205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:11.580398 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 21:39:11.582523 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:11.582529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:11.582614 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:11.582741 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:11.585187 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.586292 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.588870 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.589497 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.589656 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:11.595809 (+ 6153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:11.595815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:11.595818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:11.595854 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:11.599295 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:11.600639 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:11.600648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:11.600654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:11.600824 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:11.601368 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:11.601375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:11.601381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:11.601935 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:11.602130 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:11.602823 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 21:39:11.603118 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":342,"threads_started":1}
I20260927 21:39:11.604607 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:11.579299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40125 (local address 127.0.202.129:44649)
0927 21:39:11.580187 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:11.580238 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:11.582221 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:11.583022 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:11.583037 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:11.583917 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:11.584907 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:11.584939 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.586457 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.586464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:11.588722 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:11.588734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.589807 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.589817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:11.590164 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:11.591115 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:11.591149 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:11.595365 (+ 4216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:11.599507 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:11.599521 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:11.599546 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:11.600411 (+ 865us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:11.600953 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:11.600957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:11.600959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:11.601222 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:11.602080 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:11.602087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:11.602438 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 21:39:11.602731 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 21:39:11.603134 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":377,"threads_started":1}
I20260927 21:39:11.628077 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:11.610771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:53727 (local address 127.0.202.190:40125)
0927 21:39:11.610985 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:11.610993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:11.612135 (+ 1142us) server_negotiation.cc:407] Connection header received
0927 21:39:11.612422 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:11.612428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:11.612542 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:11.612689 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:11.613795 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.615222 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.616812 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.617665 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.617825 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:11.620154 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:11.620163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:11.620168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:11.620211 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:11.624578 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:11.625631 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:11.625641 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:11.625648 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:11.625835 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:11.626368 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:11.626373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:11.626376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:11.626745 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:11.626913 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:11.627334 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 21:39:11.627626 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:39:11.628201 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:11.610861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40125 (local address 127.0.202.129:53727)
0927 21:39:11.612019 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:11.612038 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:11.612217 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:11.612729 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:11.612734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:11.613271 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:11.613615 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:11.613627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.615430 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.615437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:11.616661 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:11.616679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:11.617888 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:11.617898 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:11.618122 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:11.619000 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:11.619027 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:11.619947 (+ 920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:11.624745 (+ 4798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:11.624753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:11.624758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:11.625479 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:11.625994 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:11.626001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:11.626010 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:11.626211 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:11.626869 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:11.626872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:11.627024 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 21:39:11.627153 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:39:11.627595 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":594,"threads_started":1}
I20260927 21:39:11.629177 2640 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41761
I20260927 21:39:11.629271 2775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41761 every 8 connection(s)
I20260927 21:39:11.630714 2640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:11.633134 2776 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40125
I20260927 21:39:11.633431 2776 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:11.634157 2776 heartbeater.cc:511] Master 127.0.202.190:40125 requested a full tablet report, sending...
I20260927 21:39:11.636402 2569 ts_manager.cc:194] Registered new tserver with Master: 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761)
I20260927 21:39:11.637861 2569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:44649
I20260927 21:39:11.641155 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 2640
I20260927 21:39:11.641335 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:11.642012 810 external_mini_cluster.cc:1428] Setting key b8af4fc4d76cb394698fd402a202c807
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:11.685348 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40125
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:12.017033 2780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:12.017338 2780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:12.017503 2780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:12.025099 2780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:12.025254 2780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:12.025377 2780 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:39:12.025436 2780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:39:12.025498 2780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:12.035255 2780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:40125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:12.036521 2780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:12.037837 2780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:12.052619 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:12.052587 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:12.052851 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:12.053387 2780 server_base.cc:1034] running on GCE node
I20260927 21:39:12.054057 2780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:12.055040 2780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:12.056269 2780 hybrid_clock.cc:648] HybridClock initialized: now 1790545152056191 us; error 61 us; skew 500 ppm
I20260927 21:39:12.056658 2780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:12 dist-test-slave-25dq krb5kdc[2524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:12.065806 2780 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:12.067462 2780 webserver.cc:460] Webserver started at http://127.0.202.130:38087/ using document root <none> and password file <none>
I20260927 21:39:12.068140 2780 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:12.068239 2780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:12.068567 2780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:12.071444 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "41c064d9af714ad7ad416798f7a14940"
format_stamp: "Formatted at 2026-09-27 21:39:12 on dist-test-slave-25dq"
server_key: "580c4d7a498232d3a20b6945bf846d6d"
server_key_iv: "1a9ea4d625943ef53eb2c5124b0020e8"
server_key_version: "encryptionkey@0"
I20260927 21:39:12.072178 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "41c064d9af714ad7ad416798f7a14940"
format_stamp: "Formatted at 2026-09-27 21:39:12 on dist-test-slave-25dq"
server_key: "580c4d7a498232d3a20b6945bf846d6d"
server_key_iv: "1a9ea4d625943ef53eb2c5124b0020e8"
server_key_version: "encryptionkey@0"
I20260927 21:39:12.077600 2780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:39:12.081224 2795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:12.083048 2780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:12.083243 2780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "41c064d9af714ad7ad416798f7a14940"
format_stamp: "Formatted at 2026-09-27 21:39:12 on dist-test-slave-25dq"
server_key: "580c4d7a498232d3a20b6945bf846d6d"
server_key_iv: "1a9ea4d625943ef53eb2c5124b0020e8"
server_key_version: "encryptionkey@0"
I20260927 21:39:12.083446 2780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:12.105679 2780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:12.110174 2780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:12.110467 2780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:12.113323 2780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:12.113446 2780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:12.113648 2780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:12.113751 2780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:39:12 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:12.137254 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.114053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:38883 (local address 127.0.202.190:40125)
0927 21:39:12.114265 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.114272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.115745 (+ 1473us) server_negotiation.cc:407] Connection header received
0927 21:39:12.118050 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.118055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.118153 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.118275 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:12.120376 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.121403 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.124123 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.124737 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.124889 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.130313 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:12.130320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:12.130324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:12.130359 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:12.133767 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.134660 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.134666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.134669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.134840 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.135421 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.135426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.135429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.135729 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:12.135885 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.136702 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.136917 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:39:12.138957 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.114764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40125 (local address 127.0.202.130:38883)
0927 21:39:12.115582 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.115626 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.117728 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.118586 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.118602 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.119489 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:12.120178 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.120196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.121592 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.121597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.123922 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.123939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.125013 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.125020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.125355 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.126343 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:12.126373 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:12.129950 (+ 3577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:12.133915 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.133924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.133942 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.134472 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.134990 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.135001 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.135007 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.135286 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.135864 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:12.135872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:12.136257 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.136628 (+ 371us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.137240 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":373,"threads_started":1}
I20260927 21:39:12.162647 2780 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:33641
I20260927 21:39:12.162669 2914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:33641 every 8 connection(s)
I20260927 21:39:12.163815 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.147126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:56005 (local address 127.0.202.190:40125)
0927 21:39:12.147296 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.147337 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.147891 (+ 554us) server_negotiation.cc:407] Connection header received
0927 21:39:12.148132 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.148138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.148261 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.148399 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:12.150038 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.151154 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.152951 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.153551 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.153711 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.156671 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:12.156680 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:12.156690 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:12.156741 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:12.160665 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.161414 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.161423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.161427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.161580 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.162115 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.162124 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.162128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.162475 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:12.162651 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.163163 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.163422 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:39:12.163954 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.147139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40125 (local address 127.0.202.130:56005)
0927 21:39:12.147778 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.147795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.147974 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.148605 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.148608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.149355 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:12.149823 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.149844 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.151311 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.151317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.152801 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.152813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.153997 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.154002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.154143 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.154999 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:12.155036 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:12.156386 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:12.160817 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.160823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.160828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.161270 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.161767 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.161775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.161781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.161974 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.162614 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:12.162623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:12.162875 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.163069 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.163446 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":302,"threads_started":1}
I20260927 21:39:12.164145 2780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:12.165441 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 2780
I20260927 21:39:12.165643 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:12.166237 810 external_mini_cluster.cc:1428] Setting key 7226675063a818f98821436f95ae4747
I20260927 21:39:12.166724 2915 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40125
I20260927 21:39:12.167547 2915 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:12.168963 2915 heartbeater.cc:511] Master 127.0.202.190:40125 requested a full tablet report, sending...
I20260927 21:39:12.170295 2569 ts_manager.cc:194] Registered new tserver with Master: 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641)
I20260927 21:39:12.171149 2569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:38883
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:12.213263 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40125
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:12.552590 2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:12.552930 2919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:12.553115 2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:12.560750 2919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:12.560913 2919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:12.561070 2919 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:39:12.561160 2919 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:39:12.561277 2919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:12.571450 2919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:40125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:12.572808 2919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:12.574188 2919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:12.587401 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:12.587517 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:12.587723 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:12.589062 2919 server_base.cc:1034] running on GCE node
I20260927 21:39:12.589787 2919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:12.590898 2919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:12.592154 2919 hybrid_clock.cc:648] HybridClock initialized: now 1790545152592107 us; error 41 us; skew 500 ppm
I20260927 21:39:12.592583 2919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:12 dist-test-slave-25dq krb5kdc[2524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:12.596359 2919 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:12.597867 2919 webserver.cc:460] Webserver started at http://127.0.202.131:37737/ using document root <none> and password file <none>
I20260927 21:39:12.598598 2919 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:12.598742 2919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:12.599056 2919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:12.602207 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef"
format_stamp: "Formatted at 2026-09-27 21:39:12 on dist-test-slave-25dq"
server_key: "52e345f77fa2a02305fa17b6ca2694a0"
server_key_iv: "7c86784e7283a06f66aace72498a1b92"
server_key_version: "encryptionkey@0"
I20260927 21:39:12.603070 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef"
format_stamp: "Formatted at 2026-09-27 21:39:12 on dist-test-slave-25dq"
server_key: "52e345f77fa2a02305fa17b6ca2694a0"
server_key_iv: "7c86784e7283a06f66aace72498a1b92"
server_key_version: "encryptionkey@0"
I20260927 21:39:12.609078 2919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:39:12.612964 2934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:12.614501 2919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:39:12.614681 2919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef"
format_stamp: "Formatted at 2026-09-27 21:39:12 on dist-test-slave-25dq"
server_key: "52e345f77fa2a02305fa17b6ca2694a0"
server_key_iv: "7c86784e7283a06f66aace72498a1b92"
server_key_version: "encryptionkey@0"
I20260927 21:39:12.614948 2919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:12.641837 2776 heartbeater.cc:503] Master 127.0.202.190:40125 was elected leader, sending a full tablet report...
I20260927 21:39:12.653668 2919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:12.660012 2919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:12.660463 2919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:12.663301 2919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:12.663420 2919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:12.663499 2919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:12.663554 2919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:39:12 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:12.690927 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.664118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:33783 (local address 127.0.202.190:40125)
0927 21:39:12.664363 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.664370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.666075 (+ 1705us) server_negotiation.cc:407] Connection header received
0927 21:39:12.669619 (+ 3544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.669628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.669723 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.669844 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:12.672370 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.673378 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.675992 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.676590 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.676749 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.682356 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:12.682361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:12.682365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:12.682429 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:12.685901 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.687273 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.687278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.687280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.687495 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.688345 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.688355 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.688360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.688815 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:12.689023 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.690366 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.690646 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:39:12.693174 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.664943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40125 (local address 127.0.202.131:33783)
0927 21:39:12.665863 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.665925 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.669142 (+ 3217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.670123 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.670132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.671055 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:12.672039 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.672062 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.673613 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.673617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.675817 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.675830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.676906 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.676921 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.677331 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.678314 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:12.678364 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:12.682083 (+ 3719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:12.686108 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.686124 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.686154 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.687087 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.687680 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.687688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.687693 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.688084 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.689015 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:12.689032 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:12.689744 (+ 712us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.690234 (+ 490us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.691010 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":349,"threads_started":1}
I20260927 21:39:12.719128 2919 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:42061
I20260927 21:39:12.719170 3053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:42061 every 8 connection(s)
I20260927 21:39:12.719301 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.701451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40125 (local address 127.0.202.131:60503)
0927 21:39:12.702569 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.702585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.702780 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.703131 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.703137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.703634 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:12.703911 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.703919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.705153 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.705157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.706289 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.706301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.707272 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.707290 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.707615 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.708968 (+ 1353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:12.709004 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:12.710590 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:12.715832 (+ 5242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.715842 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.715848 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.716506 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.717031 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:12.717037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:12.717042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:12.717218 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:12.717817 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:12.717823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:12.718086 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.718317 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.718752 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":845,"threads_started":1}
I20260927 21:39:12.720041 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.701266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:60503 (local address 127.0.202.190:40125)
0927 21:39:12.701912 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.701921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.702697 (+ 776us) server_negotiation.cc:407] Connection header received
0927 21:39:12.702911 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.702916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.702991 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.703097 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:12.704068 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.704961 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.706445 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.706938 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.707087 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.710962 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:12.710974 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:12.710983 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:12.711069 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:12.715555 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.716662 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.716669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.716673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.716854 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.717356 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.717365 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.717369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.717686 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:12.717870 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.719540 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.719809 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481}
I20260927 21:39:12.720898 2919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:12.723759 3054 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40125
I20260927 21:39:12.724056 3054 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:12.725073 3054 heartbeater.cc:511] Master 127.0.202.190:40125 requested a full tablet report, sending...
I20260927 21:39:12.726830 2569 ts_manager.cc:194] Registered new tserver with Master: af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:12.727001 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 2919
I20260927 21:39:12.727137 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:12.727553 810 external_mini_cluster.cc:1428] Setting key 78c96fdd55888a092fd03d9ce00cbe8a
I20260927 21:39:12.727965 2569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:33783
I20260927 21:39:12.730880 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:39:12 dist-test-slave-25dq krb5kdc[2524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:39:12 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:12.779937 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.765276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.202.190:40125)
0927 21:39:12.765525 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.765533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.765748 (+ 215us) server_negotiation.cc:407] Connection header received
0927 21:39:12.765972 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.765976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.766041 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.766179 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:12.767736 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.768594 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.769864 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.770298 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.770436 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.774870 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:12.774875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:12.774879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:12.774918 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:12.777511 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.778167 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.778170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.778173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.778305 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:12.778693 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:12.778696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:12.778698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:12.778958 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:12.779127 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.779442 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.779605 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 21:39:12.784825 2569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:12.793238 2569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:12.821264 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.808054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42061 (local address 127.0.0.1:57958)
0927 21:39:12.810683 (+ 2629us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.810701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.810811 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.812385 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.812392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.812534 (+ 142us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:12.813015 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.813033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.815642 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.815648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.818156 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.818186 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.820250 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.820258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.820500 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.820566 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.820682 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.820877 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2490,"thread_start_us":1344,"threads_started":1}
I20260927 21:39:12.822779 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.809036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.0.202.131:42061)
0927 21:39:12.811642 (+ 2606us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.811651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.811685 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:39:12.811787 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.811797 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.812056 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.813257 (+ 1201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:12.813649 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.815477 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.818402 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.820016 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.821940 (+ 1924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.822215 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.822289 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.822457 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2396,"thread_start_us":2181,"threads_started":1}
I20260927 21:39:12.830312 2984 tablet_service.cc:1467] Processing CreateTablet for tablet f2087454528a437ebd928d5adb107a32 (DEFAULT_TABLE table=test-table [id=394ef6dbc7a347489072ff52ded2dd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:12.832137 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.807795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.0.1:60782)
0927 21:39:12.809847 (+ 2052us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.809887 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.810069 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.811332 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.811341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.811392 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:12.811832 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.811845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.826833 (+ 14988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.826839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.829095 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.829113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.831239 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.831247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.831450 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.831489 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.831566 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.831757 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"thread_start_us":2582,"threads_started":1}
I20260927 21:39:12.832509 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2087454528a437ebd928d5adb107a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:12.833384 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.807952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.0.202.130:33641)
0927 21:39:12.810551 (+ 2599us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.810562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.810594 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:39:12.810715 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.810728 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.811056 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.812075 (+ 1019us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:12.812325 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.826629 (+ 14304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.829366 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.830966 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.832595 (+ 1629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.832833 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.832912 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.833078 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2403,"thread_start_us":2265,"threads_started":1}
I20260927 21:39:12.836858 2845 tablet_service.cc:1467] Processing CreateTablet for tablet f2087454528a437ebd928d5adb107a32 (DEFAULT_TABLE table=test-table [id=394ef6dbc7a347489072ff52ded2dd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:12.838083 2845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2087454528a437ebd928d5adb107a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:12.843024 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.811001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.0.1:33290)
0927 21:39:12.821652 (+ 10651us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.821673 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.821785 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.835565 (+ 13780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.835571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.835603 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:12.835792 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.835806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.838060 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.838066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.840148 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.840164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.842060 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.842067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.842269 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.842306 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.842465 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.842631 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10545,"thread_start_us":23259,"threads_started":1}
I20260927 21:39:12.843813 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.811304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.202.129:41761)
0927 21:39:12.834906 (+ 23602us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.834915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.834949 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:39:12.835061 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.835069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.835324 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.835984 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:12.836204 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.837837 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.840362 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.841852 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.842010 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.842226 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.843290 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.843484 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23416,"thread_start_us":24141,"threads_started":1}
I20260927 21:39:12.848147 3073 tablet_bootstrap.cc:492] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Bootstrap starting.
I20260927 21:39:12.848143 2706 tablet_service.cc:1467] Processing CreateTablet for tablet f2087454528a437ebd928d5adb107a32 (DEFAULT_TABLE table=test-table [id=394ef6dbc7a347489072ff52ded2dd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:12.851275 2706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2087454528a437ebd928d5adb107a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:12.852995 3074 tablet_bootstrap.cc:492] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Bootstrap starting.
I20260927 21:39:12.853250 3073 tablet_bootstrap.cc:654] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:12.854821 3073 log.cc:824] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:12.856312 3074 tablet_bootstrap.cc:654] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:12.858296 3073 tablet_bootstrap.cc:492] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: No bootstrap required, opened a new log
I20260927 21:39:12.858379 3074 log.cc:824] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:12.858608 3073 ts_tablet_manager.cc:1397] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 21:39:12.863469 3074 tablet_bootstrap.cc:492] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: No bootstrap required, opened a new log
I20260927 21:39:12.863765 3074 ts_tablet_manager.cc:1397] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260927 21:39:12.866113 3077 tablet_bootstrap.cc:492] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Bootstrap starting.
I20260927 21:39:12.869323 3077 tablet_bootstrap.cc:654] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:12.869354 3073 raft_consensus.cc:348] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.869841 3073 raft_consensus.cc:374] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:12.869956 3073 raft_consensus.cc:729] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7ddd86f79b4977adb6fe6f8c98a9ef, State: Initialized, Role: FOLLOWER
I20260927 21:39:12.870600 3077 log.cc:824] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:12.870609 3073 consensus_queue.cc:260] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.871918 3054 heartbeater.cc:503] Master 127.0.202.190:40125 was elected leader, sending a full tablet report...
I20260927 21:39:12.872478 3073 ts_tablet_manager.cc:1428] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260927 21:39:12.874818 3074 raft_consensus.cc:348] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.875190 3074 raft_consensus.cc:374] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:12.875319 3074 raft_consensus.cc:729] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Initialized, Role: FOLLOWER
I20260927 21:39:12.876083 3077 tablet_bootstrap.cc:492] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: No bootstrap required, opened a new log
I20260927 21:39:12.875905 3074 consensus_queue.cc:260] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.876416 3077 ts_tablet_manager.cc:1397] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 21:39:12.877043 2915 heartbeater.cc:503] Master 127.0.202.190:40125 was elected leader, sending a full tablet report...
I20260927 21:39:12.877568 3074 ts_tablet_manager.cc:1428] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260927 21:39:12.884721 3077 raft_consensus.cc:348] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.885087 3077 raft_consensus.cc:374] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:12.885159 3077 raft_consensus.cc:729] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03525bbaa2c04165a87dba22117d5868, State: Initialized, Role: FOLLOWER
I20260927 21:39:12.885774 3077 consensus_queue.cc:260] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.887734 3077 ts_tablet_manager.cc:1428] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260927 21:39:12.918740 2916 tablet.cc:2369] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:12.924083 3081 raft_consensus.cc:482] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:12.924468 3081 raft_consensus.cc:504] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.926638 3081 leader_election.cc:288] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:12.934383 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.927395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:58179 (local address 127.0.202.131:42061)
0927 21:39:12.927652 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.927660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.928181 (+ 521us) server_negotiation.cc:407] Connection header received
0927 21:39:12.928405 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.928411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.928501 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.928597 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:12.929330 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.930206 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.932368 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.933319 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.933494 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.933596 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.933940 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.934106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:39:12.934391 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.927057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42061 (local address 127.0.202.129:58179)
0927 21:39:12.928013 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.928035 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.928204 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.928647 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.928659 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.928690 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:12.929140 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.929151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.930336 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.930341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.932149 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.932160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.933568 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.933585 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.933777 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.933801 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.933875 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.934002 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":527,"threads_started":1}
I20260927 21:39:12.934439 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.927412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:47277 (local address 127.0.202.130:33641)
0927 21:39:12.927654 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.927661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.928363 (+ 702us) server_negotiation.cc:407] Connection header received
0927 21:39:12.928708 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.928716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.928810 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.928920 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:12.929692 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.930523 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.932560 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.933366 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.933569 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.933649 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:12.934060 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 21:39:12.934196 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:39:12.934573 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.927234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.202.129:47277)
0927 21:39:12.928205 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.928224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.928421 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.928955 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.928959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.928978 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:12.929457 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.929470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.930694 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.930702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.932391 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.932405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.933663 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.933669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:12.933845 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:12.933909 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 21:39:12.934010 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:39:12.934154 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":42,"thread_start_us":615,"threads_started":1}
I20260927 21:39:12.935772 2865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2087454528a437ebd928d5adb107a32" candidate_uuid: "03525bbaa2c04165a87dba22117d5868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c064d9af714ad7ad416798f7a14940" is_pre_election: true
I20260927 21:39:12.935870 3004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2087454528a437ebd928d5adb107a32" candidate_uuid: "03525bbaa2c04165a87dba22117d5868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" is_pre_election: true
I20260927 21:39:12.936393 2865 raft_consensus.cc:2454] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03525bbaa2c04165a87dba22117d5868 in term 0.
I20260927 21:39:12.936623 3004 raft_consensus.cc:2454] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03525bbaa2c04165a87dba22117d5868 in term 0.
I20260927 21:39:12.937237 2658 leader_election.cc:302] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03525bbaa2c04165a87dba22117d5868, 41c064d9af714ad7ad416798f7a14940; no voters:
I20260927 21:39:12.937697 3081 raft_consensus.cc:2781] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:12.937855 3081 raft_consensus.cc:482] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:12.937964 3081 raft_consensus.cc:3037] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:12.940243 3081 raft_consensus.cc:504] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.940996 3081 leader_election.cc:288] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [CANDIDATE]: Term 1 election: Requested vote from peers 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:12.941398 2865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2087454528a437ebd928d5adb107a32" candidate_uuid: "03525bbaa2c04165a87dba22117d5868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c064d9af714ad7ad416798f7a14940"
I20260927 21:39:12.941535 3004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2087454528a437ebd928d5adb107a32" candidate_uuid: "03525bbaa2c04165a87dba22117d5868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef"
I20260927 21:39:12.941624 2865 raft_consensus.cc:3037] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:12.941761 3004 raft_consensus.cc:3037] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:12.944010 3004 raft_consensus.cc:2454] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03525bbaa2c04165a87dba22117d5868 in term 1.
I20260927 21:39:12.944104 2865 raft_consensus.cc:2454] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03525bbaa2c04165a87dba22117d5868 in term 1.
I20260927 21:39:12.944485 2659 leader_election.cc:302] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03525bbaa2c04165a87dba22117d5868, af7ddd86f79b4977adb6fe6f8c98a9ef; no voters:
I20260927 21:39:12.944836 3081 raft_consensus.cc:2781] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:12.945312 3081 raft_consensus.cc:686] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 1 LEADER]: Becoming Leader. State: Replica: 03525bbaa2c04165a87dba22117d5868, State: Running, Role: LEADER
I20260927 21:39:12.945777 3081 consensus_queue.cc:237] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } }
I20260927 21:39:12.950799 2569 catalog_manager.cc:5495] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03525bbaa2c04165a87dba22117d5868 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "03525bbaa2c04165a87dba22117d5868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } health_report { overall_health: HEALTHY } } }
W20260927 21:39:12.972856 3055 tablet.cc:2369] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:39:12 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:13.010893 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.995581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.0.202.190:40125)
0927 21:39:12.995933 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:12.995942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:12.996628 (+ 686us) server_negotiation.cc:407] Connection header received
0927 21:39:12.996788 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:12.996791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:12.996878 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:12.996965 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:12.998324 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.999283 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.000737 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.001161 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.001265 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.004735 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:13.004740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:13.004744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:13.004776 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:13.007874 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:13.008789 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:13.008798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:13.008800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:13.008923 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:13.009315 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:13.009320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:13.009329 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:13.009630 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:13.009785 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.010191 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.010513 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":106}
I20260927 21:39:13.010896 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:12.995598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40125 (local address 127.0.0.1:38378)
0927 21:39:12.996516 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:12.996534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:12.996669 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:12.997038 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:12.997041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:12.997835 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:12.998097 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:12.998105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:12.999427 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:12.999431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.000516 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.000527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.001341 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.001346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.001469 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.002140 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:13.002168 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:13.004516 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:13.008067 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:13.008085 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:13.008094 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:13.008626 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:13.009035 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:13.009042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:13.009046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:13.009197 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:13.009771 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:13.009779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:13.009937 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 21:39:13.010092 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 21:39:13.010395 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":557,"threads_started":1}
I20260927 21:39:13.017182 2569 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:38378:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:39:13.018378 2569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:13.026420 2706 tablet_service.cc:1467] Processing CreateTablet for tablet 3785198da1834e10983ed851af9e1691 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=791b6eb283a14b799d14580bdd1ae154]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:39:13.026543 2845 tablet_service.cc:1467] Processing CreateTablet for tablet 3785198da1834e10983ed851af9e1691 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=791b6eb283a14b799d14580bdd1ae154]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:39:13.026948 2984 tablet_service.cc:1467] Processing CreateTablet for tablet 3785198da1834e10983ed851af9e1691 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=791b6eb283a14b799d14580bdd1ae154]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:39:13.027503 2845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3785198da1834e10983ed851af9e1691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:13.027484 2706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3785198da1834e10983ed851af9e1691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:13.027791 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3785198da1834e10983ed851af9e1691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:13.033382 3077 tablet_bootstrap.cc:492] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: Bootstrap starting.
I20260927 21:39:13.034619 3077 tablet_bootstrap.cc:654] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:13.035652 3073 tablet_bootstrap.cc:492] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: Bootstrap starting.
I20260927 21:39:13.036219 3074 tablet_bootstrap.cc:492] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: Bootstrap starting.
I20260927 21:39:13.037396 3077 tablet_bootstrap.cc:492] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: No bootstrap required, opened a new log
I20260927 21:39:13.037477 3073 tablet_bootstrap.cc:654] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:13.037569 3077 ts_tablet_manager.cc:1397] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:39:13.038143 3074 tablet_bootstrap.cc:654] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:13.038256 3077 raft_consensus.cc:348] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.038493 3077 raft_consensus.cc:374] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:13.038594 3077 raft_consensus.cc:729] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03525bbaa2c04165a87dba22117d5868, State: Initialized, Role: FOLLOWER
I20260927 21:39:13.039002 3077 consensus_queue.cc:260] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.039494 3081 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.039841 3081 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:13.040926 3077 ts_tablet_manager.cc:1428] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:13.041018 3073 tablet_bootstrap.cc:492] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: No bootstrap required, opened a new log
I20260927 21:39:13.041258 3073 ts_tablet_manager.cc:1397] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 21:39:13.042552 3073 raft_consensus.cc:348] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.042856 3073 raft_consensus.cc:374] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:13.042896 3074 tablet_bootstrap.cc:492] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: No bootstrap required, opened a new log
I20260927 21:39:13.042973 3073 raft_consensus.cc:729] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7ddd86f79b4977adb6fe6f8c98a9ef, State: Initialized, Role: FOLLOWER
I20260927 21:39:13.043068 3074 ts_tablet_manager.cc:1397] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 21:39:13.043304 3073 consensus_queue.cc:260] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.043859 3078 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.043891 3074 raft_consensus.cc:348] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.044337 3074 raft_consensus.cc:374] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:13.044437 3078 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:13.044481 3074 raft_consensus.cc:729] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Initialized, Role: FOLLOWER
I20260927 21:39:13.044965 3074 consensus_queue.cc:260] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.045272 3073 ts_tablet_manager.cc:1428] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260927 21:39:13.045534 3080 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.045960 3080 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:13.046540 3074 ts_tablet_manager.cc:1428] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260927 21:39:13.134996 2777 tablet.cc:2369] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:13.248080 3080 raft_consensus.cc:482] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:13.248396 3080 raft_consensus.cc:504] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.250177 3080 leader_election.cc:288] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:13.260612 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.250672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:55953 (local address 127.0.202.131:42061)
0927 21:39:13.251006 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.251015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.252224 (+ 1209us) server_negotiation.cc:407] Connection header received
0927 21:39:13.252398 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.252404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.252473 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.252566 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:13.253733 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.255051 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.258434 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.259466 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.259576 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.259634 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.260144 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.260323 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260927 21:39:13.260689 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.250572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:46075 (local address 127.0.202.129:41761)
0927 21:39:13.250834 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.250845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.251970 (+ 1125us) server_negotiation.cc:407] Connection header received
0927 21:39:13.252390 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.252395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.252473 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.252574 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:13.253838 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.255074 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.258395 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.259596 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.259791 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.259882 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.260240 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.260406 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:39:13.260831 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.250627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.202.130:46075)
0927 21:39:13.251841 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:13.251861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:13.252104 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:13.252691 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:13.252697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:13.252768 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:13.253459 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.253480 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.255260 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.255268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.258143 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.258161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.259880 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.259888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.260051 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.260074 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:39:13.260191 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 21:39:13.260385 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":126,"spinlock_wait_cycles":13696,"thread_start_us":581,"threads_started":1}
I20260927 21:39:13.261193 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.250768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42061 (local address 127.0.202.130:55953)
0927 21:39:13.252099 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:13.252117 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:13.252235 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:13.252691 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:13.252697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:13.252766 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:13.253371 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.253387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.255271 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.255280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.258145 (+ 2865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.258204 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.259828 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.259834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.259971 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.259996 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:39:13.260118 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 21:39:13.260318 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"mutex_wait_us":35,"spinlock_wait_cycles":35072,"thread_start_us":943,"threads_started":1}
I20260927 21:39:13.261793 3004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3785198da1834e10983ed851af9e1691" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" is_pre_election: true
I20260927 21:39:13.261808 2726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3785198da1834e10983ed851af9e1691" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868" is_pre_election: true
I20260927 21:39:13.262064 3004 raft_consensus.cc:2454] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 0.
I20260927 21:39:13.262298 2726 raft_consensus.cc:2454] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 0.
I20260927 21:39:13.262588 2799 leader_election.cc:302] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c064d9af714ad7ad416798f7a14940, af7ddd86f79b4977adb6fe6f8c98a9ef; no voters:
I20260927 21:39:13.263082 3080 raft_consensus.cc:2781] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:13.263233 3080 raft_consensus.cc:482] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:13.263334 3080 raft_consensus.cc:3037] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:13.265897 3080 raft_consensus.cc:504] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.266656 3080 leader_election.cc:288] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 election: Requested vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:13.267204 2726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3785198da1834e10983ed851af9e1691" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868"
I20260927 21:39:13.267228 3004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3785198da1834e10983ed851af9e1691" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef"
I20260927 21:39:13.267405 2726 raft_consensus.cc:3037] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:13.267448 3004 raft_consensus.cc:3037] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:13.269567 3004 raft_consensus.cc:2454] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 1.
I20260927 21:39:13.269865 2726 raft_consensus.cc:2454] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 1.
I20260927 21:39:13.270069 2799 leader_election.cc:302] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c064d9af714ad7ad416798f7a14940, af7ddd86f79b4977adb6fe6f8c98a9ef; no voters:
I20260927 21:39:13.270427 3080 raft_consensus.cc:2781] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:13.270888 3080 raft_consensus.cc:686] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [term 1 LEADER]: Becoming Leader. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Running, Role: LEADER
I20260927 21:39:13.271373 3080 consensus_queue.cc:237] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.274811 3097 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: TxnStatusTablet state changed. Reason: New leader 41c064d9af714ad7ad416798f7a14940. Latest consensus state: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.275251 3097 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:39:13.276104 2569 catalog_manager.cc:5495] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c064d9af714ad7ad416798f7a14940 (127.0.202.130). New cstate: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } health_report { overall_health: UNKNOWN } } }
I20260927 21:39:13.277333 3099 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:39:13.277617 3099 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:39:13.340014 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.333409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.0.202.130:33641)
0927 21:39:13.333638 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.333643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.333753 (+ 110us) server_negotiation.cc:407] Connection header received
0927 21:39:13.334107 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.334112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.334195 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.334365 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:13.334812 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.335699 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.337707 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.338196 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.338344 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.338849 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.339431 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.339549 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.339680 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.339756 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:39:13.340132 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.333347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.0.1:60790)
0927 21:39:13.333646 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:13.333668 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:13.333827 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:13.334383 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:13.334388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:13.334401 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:13.334624 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.334636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.335858 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.335864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.337494 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.337509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.338361 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.338368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.338582 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.338658 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.339547 (+ 889us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.339570 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:39:13.339661 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:39:13.339822 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 21:39:13.349407 2726 raft_consensus.cc:1261] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Refusing update from remote peer 41c064d9af714ad7ad416798f7a14940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:13.349537 3004 raft_consensus.cc:1261] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Refusing update from remote peer 41c064d9af714ad7ad416798f7a14940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:13.350567 3097 consensus_queue.cc:1035] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:13.351018 3080 consensus_queue.cc:1035] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:13.363397 3081 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: TxnStatusTablet state changed. Reason: New leader 41c064d9af714ad7ad416798f7a14940. Latest consensus state: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.363713 3081 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:13.364317 3081 consensus_queue.cc:1035] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:13.367278 3078 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: TxnStatusTablet state changed. Reason: New leader 41c064d9af714ad7ad416798f7a14940. Latest consensus state: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.367586 3078 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:13.368245 3102 mvcc.cc:205] Tried to move back new op lower bound from 7334072948108091392 to 7334072947803504640. Current Snapshot: MvccSnapshot[applied={T|T < 7334072948108091392}]
I20260927 21:39:13.371424 3104 mvcc.cc:205] Tried to move back new op lower bound from 7334072948108091392 to 7334072947803504640. Current Snapshot: MvccSnapshot[applied={T|T < 7334072948108091392}]
I20260927 21:39:13.376811 3080 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.377118 3080 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:39:13.377420 3080 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.377629 3080 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P 41c064d9af714ad7ad416798f7a14940: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:39:13.382645 3081 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.382890 3081 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P 03525bbaa2c04165a87dba22117d5868: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:13.390213 3081 consensus_queue.cc:1035] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:13.399072 3078 tablet_replica.cc:439] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } } }
I20260927 21:39:13.399361 3078 tablet_replica.cc:442] T 3785198da1834e10983ed851af9e1691 P af7ddd86f79b4977adb6fe6f8c98a9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:39:13.417421 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.400716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.0.202.129:41761)
0927 21:39:13.400902 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.400911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.409973 (+ 9062us) server_negotiation.cc:407] Connection header received
0927 21:39:13.410256 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.410263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.410378 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.410542 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:13.411255 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.412629 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.414765 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.415402 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.415530 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.416000 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.416791 (+ 791us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.416933 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.417004 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.417090 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:39:13.433952 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.428412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:48865 (local address 127.0.202.130:33641)
0927 21:39:13.428602 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.428610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.429121 (+ 511us) server_negotiation.cc:407] Connection header received
0927 21:39:13.429341 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.429345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.429408 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.429508 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:13.429998 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.430762 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.432106 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.432632 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.432779 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.433202 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.433303 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.433397 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.433539 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.433616 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:39:13.434048 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.428398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.202.129:48865)
0927 21:39:13.429019 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:13.429034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:13.429188 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:13.429520 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:13.429524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:13.429536 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:13.429844 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.429852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.430896 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.430903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.431970 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.431980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.432812 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.432819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.432953 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.433003 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.433399 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.433418 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 21:39:13.433516 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:39:13.433644 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":308,"threads_started":1}
I20260927 21:39:13.476181 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.468491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:44947 (local address 127.0.202.129:41761)
0927 21:39:13.468685 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.468692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.469486 (+ 794us) server_negotiation.cc:407] Connection header received
0927 21:39:13.469793 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.469800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.469900 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.470008 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:13.470671 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.471762 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.473985 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.474637 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.474731 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.475393 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.475524 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.475584 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.475805 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.475887 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:39:13.477005 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.468683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.202.130:44947)
0927 21:39:13.469394 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:13.469413 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:13.469617 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:13.470029 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:13.470034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:13.470049 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:13.470489 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.470501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.471934 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.471939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.473822 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.473842 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.474917 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.474923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.475119 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.475202 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.475661 (+ 459us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.475671 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:39:13.476446 (+ 775us) client_negotiation.cc:240] Negotiation successful
0927 21:39:13.476588 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":340,"threads_started":1}
I20260927 21:39:13.515027 2568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:13.515327 2568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:13.519447 2568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Sending DeleteTablet for 3 replicas of tablet f2087454528a437ebd928d5adb107a32
I20260927 21:39:13.520834 2984 tablet_service.cc:1514] Processing DeleteTablet for tablet f2087454528a437ebd928d5adb107a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:13 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:57958
I20260927 21:39:13.521464 2706 tablet_service.cc:1514] Processing DeleteTablet for tablet f2087454528a437ebd928d5adb107a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:13 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:33290
I20260927 21:39:13.523985 2567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:13.524953 2567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:13.535290 2844 tablet_service.cc:1467] Processing CreateTablet for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 (DEFAULT_TABLE table=test-table [id=60c2ae026f9c44d3a4f0ee178015d458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:13.536047 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:13.541393 2983 tablet_service.cc:1467] Processing CreateTablet for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 (DEFAULT_TABLE table=test-table [id=60c2ae026f9c44d3a4f0ee178015d458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:13.541936 2983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:13.545658 2705 tablet_service.cc:1467] Processing CreateTablet for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 (DEFAULT_TABLE table=test-table [id=60c2ae026f9c44d3a4f0ee178015d458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:13.546151 2705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:13.552910 2845 tablet_service.cc:1514] Processing DeleteTablet for tablet f2087454528a437ebd928d5adb107a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:13 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60782
I20260927 21:39:13.558601 3127 tablet_replica.cc:330] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: stopping tablet replica
I20260927 21:39:13.560127 3127 raft_consensus.cc:2229] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:13.560539 3127 raft_consensus.cc:2258] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:13.564270 3127 ts_tablet_manager.cc:1905] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:13.572885 3128 tablet_bootstrap.cc:492] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: Bootstrap starting.
I20260927 21:39:13.575608 3128 tablet_bootstrap.cc:654] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:13.577391 3126 tablet_bootstrap.cc:492] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: Bootstrap starting.
I20260927 21:39:13.579876 3126 tablet_bootstrap.cc:654] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:13.580339 3125 tablet_replica.cc:330] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: stopping tablet replica
I20260927 21:39:13.580824 3125 raft_consensus.cc:2229] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:13.581508 3125 raft_consensus.cc:2258] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:13.582409 3129 tablet_bootstrap.cc:492] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: Bootstrap starting.
I20260927 21:39:13.588336 3129 tablet_bootstrap.cc:654] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:13.590337 3125 ts_tablet_manager.cc:1905] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:13.595721 3124 tablet_replica.cc:330] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: stopping tablet replica
I20260927 21:39:13.596340 3124 raft_consensus.cc:2229] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:13.596835 3124 raft_consensus.cc:2258] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:13.599123 3124 ts_tablet_manager.cc:1905] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:13.605115 3126 tablet_bootstrap.cc:492] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: No bootstrap required, opened a new log
I20260927 21:39:13.605310 3126 ts_tablet_manager.cc:1397] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.005s
I20260927 21:39:13.606191 3126 raft_consensus.cc:348] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.608204 3127 ts_tablet_manager.cc:1918] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:13.606555 3126 raft_consensus.cc:374] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:13.608649 3126 raft_consensus.cc:729] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03525bbaa2c04165a87dba22117d5868, State: Initialized, Role: FOLLOWER
I20260927 21:39:13.608939 3126 consensus_queue.cc:260] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.610543 3128 tablet_bootstrap.cc:492] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: No bootstrap required, opened a new log
I20260927 21:39:13.610777 3128 ts_tablet_manager.cc:1397] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.000s
I20260927 21:39:13.611790 3128 raft_consensus.cc:348] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.612161 3128 raft_consensus.cc:374] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:13.612355 3128 raft_consensus.cc:729] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7ddd86f79b4977adb6fe6f8c98a9ef, State: Initialized, Role: FOLLOWER
I20260927 21:39:13.612421 3124 ts_tablet_manager.cc:1918] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:13.612598 3124 log.cc:1196] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/f2087454528a437ebd928d5adb107a32
I20260927 21:39:13.612865 3128 consensus_queue.cc:260] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.613186 3124 ts_tablet_manager.cc:1939] T f2087454528a437ebd928d5adb107a32 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting consensus metadata
I20260927 21:39:13.613219 3125 ts_tablet_manager.cc:1918] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:13.613368 3125 log.cc:1196] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/f2087454528a437ebd928d5adb107a32
I20260927 21:39:13.613920 3125 ts_tablet_manager.cc:1939] T f2087454528a437ebd928d5adb107a32 P 03525bbaa2c04165a87dba22117d5868: Deleting consensus metadata
I20260927 21:39:13.615279 2556 catalog_manager.cc:4841] TS af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061): tablet f2087454528a437ebd928d5adb107a32 (table test-table [id=394ef6dbc7a347489072ff52ded2dd58]) successfully deleted
I20260927 21:39:13.615746 3127 log.cc:1196] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/f2087454528a437ebd928d5adb107a32
I20260927 21:39:13.617300 3126 ts_tablet_manager.cc:1428] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20260927 21:39:13.618371 3129 tablet_bootstrap.cc:492] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: No bootstrap required, opened a new log
I20260927 21:39:13.618527 3129 ts_tablet_manager.cc:1397] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s
I20260927 21:39:13.619311 3127 ts_tablet_manager.cc:1939] T f2087454528a437ebd928d5adb107a32 P 41c064d9af714ad7ad416798f7a14940: Deleting consensus metadata
I20260927 21:39:13.621040 2555 catalog_manager.cc:4841] TS 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641): tablet f2087454528a437ebd928d5adb107a32 (table test-table [id=394ef6dbc7a347489072ff52ded2dd58]) successfully deleted
I20260927 21:39:13.621240 2553 catalog_manager.cc:4841] TS 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761): tablet f2087454528a437ebd928d5adb107a32 (table test-table [id=394ef6dbc7a347489072ff52ded2dd58]) successfully deleted
I20260927 21:39:13.621445 3129 raft_consensus.cc:348] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.621729 3129 raft_consensus.cc:374] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:13.621825 3129 raft_consensus.cc:729] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Initialized, Role: FOLLOWER
I20260927 21:39:13.622175 3129 consensus_queue.cc:260] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.622882 3129 ts_tablet_manager.cc:1428] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:39:13.625085 3128 ts_tablet_manager.cc:1428] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20260927 21:39:13.630203 3097 raft_consensus.cc:482] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:13.630421 3097 raft_consensus.cc:504] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.631232 3097 leader_election.cc:288] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:13.632516 2726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f4a44fe3c14bd2944c5ed8d4ac4bf9" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868" is_pre_election: true
I20260927 21:39:13.632807 2726 raft_consensus.cc:2454] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 0.
I20260927 21:39:13.633203 3003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f4a44fe3c14bd2944c5ed8d4ac4bf9" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" is_pre_election: true
I20260927 21:39:13.633440 3003 raft_consensus.cc:2454] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 0.
I20260927 21:39:13.634171 2799 leader_election.cc:302] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c064d9af714ad7ad416798f7a14940, af7ddd86f79b4977adb6fe6f8c98a9ef; no voters:
I20260927 21:39:13.634932 3097 raft_consensus.cc:2781] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:13.635041 3097 raft_consensus.cc:482] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:13.635169 3097 raft_consensus.cc:3037] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:13.638489 3097 raft_consensus.cc:504] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.639708 2726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f4a44fe3c14bd2944c5ed8d4ac4bf9" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868"
I20260927 21:39:13.639954 2726 raft_consensus.cc:3037] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:13.643607 2726 raft_consensus.cc:2454] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 1.
I20260927 21:39:13.644177 2796 leader_election.cc:302] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03525bbaa2c04165a87dba22117d5868, 41c064d9af714ad7ad416798f7a14940; no voters:
I20260927 21:39:13.644584 3080 raft_consensus.cc:2781] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:13.644760 3080 raft_consensus.cc:686] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 1 LEADER]: Becoming Leader. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Running, Role: LEADER
I20260927 21:39:13.645071 3080 consensus_queue.cc:237] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:13.648639 3003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f4a44fe3c14bd2944c5ed8d4ac4bf9" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef"
I20260927 21:39:13.648937 3003 raft_consensus.cc:3037] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:13.652396 3003 raft_consensus.cc:2454] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 1.
I20260927 21:39:13.653326 3097 leader_election.cc:288] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 election: Requested vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:13.662096 2568 catalog_manager.cc:5495] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c064d9af714ad7ad416798f7a14940 (127.0.202.130). New cstate: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } health_report { overall_health: UNKNOWN } } }
I20260927 21:39:13.709712 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.701836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.0.202.130:33641)
0927 21:39:13.702384 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.702394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.702419 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:39:13.702549 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.702557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.702661 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.702783 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:13.703208 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.704737 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.706744 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.707587 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.707856 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.708388 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.708792 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.708901 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.709068 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.709170 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260927 21:39:13.732357 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.720585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.202.130:55487)
0927 21:39:13.720781 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:13.720802 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:13.720914 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:13.725351 (+ 4437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:13.725356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:13.725370 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:13.725610 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.725624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.727861 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.727867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.729729 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:13.729747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.730887 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.730894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:13.731107 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.731163 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.731702 (+ 539us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.731710 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:39:13.731791 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 21:39:13.731965 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 21:39:13.733186 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:13.724554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:55487 (local address 127.0.202.130:33641)
0927 21:39:13.724762 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:13.724770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:13.724794 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:39:13.724873 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:13.724879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:13.724977 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:13.725114 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:13.725858 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.727196 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.729925 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:13.730626 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:13.730768 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:13.731369 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:13.731534 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:13.731657 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:13.732792 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0927 21:39:13.732872 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:39:13.745137 3004 raft_consensus.cc:1261] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Refusing update from remote peer 41c064d9af714ad7ad416798f7a14940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:13.745865 2725 raft_consensus.cc:1261] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Refusing update from remote peer 41c064d9af714ad7ad416798f7a14940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:13.745851 3119 consensus_queue.cc:1035] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:13.748639 3097 consensus_queue.cc:1035] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:13.764376 3103 mvcc.cc:205] Tried to move back new op lower bound from 7334072949734658048 to 7334072949333250048. Current Snapshot: MvccSnapshot[applied={T|T < 7334072949734658048}]
W20260927 21:39:13.795084 3102 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:39:15.400537 2568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:15.400964 2568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:15.405479 2568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Sending DeleteTablet for 3 replicas of tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9
I20260927 21:39:15.406514 2844 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:15 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60782
I20260927 21:39:15.406512 2706 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:15 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:33290
I20260927 21:39:15.406674 2984 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:15 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:57958
I20260927 21:39:15.419474 3158 tablet_replica.cc:330] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: stopping tablet replica
I20260927 21:39:15.420341 3158 raft_consensus.cc:2229] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:15.420754 3158 raft_consensus.cc:2258] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:15.421165 3157 tablet_replica.cc:330] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: stopping tablet replica
I20260927 21:39:15.421446 3157 raft_consensus.cc:2229] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:15.422210 3157 raft_consensus.cc:2258] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:15.422264 3158 ts_tablet_manager.cc:1905] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:15.423321 3157 ts_tablet_manager.cc:1905] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:15.424757 3159 tablet_replica.cc:330] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: stopping tablet replica
I20260927 21:39:15.425026 3159 raft_consensus.cc:2229] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:15.425318 3159 raft_consensus.cc:2258] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:15.426344 3159 ts_tablet_manager.cc:1905] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:15.429687 3158 ts_tablet_manager.cc:1918] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:39:15.429900 3158 log.cc:1196] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/d8f4a44fe3c14bd2944c5ed8d4ac4bf9
I20260927 21:39:15.430012 3157 ts_tablet_manager.cc:1918] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:39:15.430178 3157 log.cc:1196] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/d8f4a44fe3c14bd2944c5ed8d4ac4bf9
I20260927 21:39:15.430428 3158 ts_tablet_manager.cc:1939] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 03525bbaa2c04165a87dba22117d5868: Deleting consensus metadata
I20260927 21:39:15.430721 3157 ts_tablet_manager.cc:1939] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P 41c064d9af714ad7ad416798f7a14940: Deleting consensus metadata
I20260927 21:39:15.431774 2555 catalog_manager.cc:4841] TS 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641): tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 (table test-table [id=60c2ae026f9c44d3a4f0ee178015d458]) successfully deleted
I20260927 21:39:15.431783 2553 catalog_manager.cc:4841] TS 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761): tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 (table test-table [id=60c2ae026f9c44d3a4f0ee178015d458]) successfully deleted
I20260927 21:39:15.433630 3159 ts_tablet_manager.cc:1918] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:39:15.433795 3159 log.cc:1196] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/d8f4a44fe3c14bd2944c5ed8d4ac4bf9
I20260927 21:39:15.434223 3159 ts_tablet_manager.cc:1939] T d8f4a44fe3c14bd2944c5ed8d4ac4bf9 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting consensus metadata
I20260927 21:39:15.435181 2556 catalog_manager.cc:4841] TS af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061): tablet d8f4a44fe3c14bd2944c5ed8d4ac4bf9 (table test-table [id=60c2ae026f9c44d3a4f0ee178015d458]) successfully deleted
Sep 27 21:39:15 dist-test-slave-25dq krb5kdc[2524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:39:15 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:15.472390 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.456611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.202.190:40125)
0927 21:39:15.457844 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:15.457850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:15.457878 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:39:15.458035 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:15.458039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:15.458104 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:15.458226 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:15.459699 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.460560 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.461850 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.462275 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.462416 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.466417 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:15.466426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:15.466433 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:15.466486 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:15.469498 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:15.470170 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:15.470174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:15.470176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:15.470313 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:15.470770 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:15.470774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:15.470776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:15.471100 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:15.471296 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:15.471728 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 21:39:15.471909 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":899,"threads_started":1}
I20260927 21:39:15.476554 2568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:15.477134 2568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:15.485111 2706 tablet_service.cc:1467] Processing CreateTablet for tablet 95a1d267f0ef4820afdc0ee79d1e9b41 (DEFAULT_TABLE table=test-table [id=c57a9a30773143a3ade917ce16d6d1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:15.485098 2844 tablet_service.cc:1467] Processing CreateTablet for tablet 95a1d267f0ef4820afdc0ee79d1e9b41 (DEFAULT_TABLE table=test-table [id=c57a9a30773143a3ade917ce16d6d1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:15.485270 2984 tablet_service.cc:1467] Processing CreateTablet for tablet 95a1d267f0ef4820afdc0ee79d1e9b41 (DEFAULT_TABLE table=test-table [id=c57a9a30773143a3ade917ce16d6d1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:15.485613 2706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a1d267f0ef4820afdc0ee79d1e9b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:15.485634 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a1d267f0ef4820afdc0ee79d1e9b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:15.485723 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a1d267f0ef4820afdc0ee79d1e9b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:15.491989 3168 tablet_bootstrap.cc:492] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: Bootstrap starting.
I20260927 21:39:15.492832 3170 tablet_bootstrap.cc:492] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: Bootstrap starting.
I20260927 21:39:15.493654 3169 tablet_bootstrap.cc:492] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: Bootstrap starting.
I20260927 21:39:15.494114 3168 tablet_bootstrap.cc:654] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:15.494621 3170 tablet_bootstrap.cc:654] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:15.495400 3169 tablet_bootstrap.cc:654] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:15.497339 3168 tablet_bootstrap.cc:492] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: No bootstrap required, opened a new log
I20260927 21:39:15.497531 3168 ts_tablet_manager.cc:1397] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 21:39:15.497696 3170 tablet_bootstrap.cc:492] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: No bootstrap required, opened a new log
I20260927 21:39:15.497884 3170 ts_tablet_manager.cc:1397] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 21:39:15.498495 3168 raft_consensus.cc:348] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.498757 3169 tablet_bootstrap.cc:492] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: No bootstrap required, opened a new log
I20260927 21:39:15.498806 3168 raft_consensus.cc:374] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:15.498912 3169 ts_tablet_manager.cc:1397] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260927 21:39:15.498904 3168 raft_consensus.cc:729] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7ddd86f79b4977adb6fe6f8c98a9ef, State: Initialized, Role: FOLLOWER
I20260927 21:39:15.498792 3170 raft_consensus.cc:348] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.499037 3170 raft_consensus.cc:374] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:15.499128 3170 raft_consensus.cc:729] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Initialized, Role: FOLLOWER
I20260927 21:39:15.499183 3168 consensus_queue.cc:260] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.499408 3170 consensus_queue.cc:260] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.499922 3169 raft_consensus.cc:348] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.500442 3169 raft_consensus.cc:374] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:15.500624 3169 raft_consensus.cc:729] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03525bbaa2c04165a87dba22117d5868, State: Initialized, Role: FOLLOWER
I20260927 21:39:15.500648 3170 ts_tablet_manager.cc:1428] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:39:15.501036 3169 consensus_queue.cc:260] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.502372 3169 ts_tablet_manager.cc:1428] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:39:15.509104 3168 ts_tablet_manager.cc:1428] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260927 21:39:15.568908 3080 raft_consensus.cc:482] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:15.569164 3080 raft_consensus.cc:504] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.569759 3080 leader_election.cc:288] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:15.570508 2725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a1d267f0ef4820afdc0ee79d1e9b41" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868" is_pre_election: true
I20260927 21:39:15.570528 3004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a1d267f0ef4820afdc0ee79d1e9b41" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" is_pre_election: true
I20260927 21:39:15.570775 3004 raft_consensus.cc:2454] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 0.
I20260927 21:39:15.570789 2725 raft_consensus.cc:2454] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 0.
I20260927 21:39:15.571265 2799 leader_election.cc:302] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c064d9af714ad7ad416798f7a14940, af7ddd86f79b4977adb6fe6f8c98a9ef; no voters:
I20260927 21:39:15.571573 3080 raft_consensus.cc:2781] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:15.571676 3080 raft_consensus.cc:482] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:15.571786 3080 raft_consensus.cc:3037] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:15.573932 3080 raft_consensus.cc:504] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.574539 3080 leader_election.cc:288] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 election: Requested vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061)
I20260927 21:39:15.575151 2725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a1d267f0ef4820afdc0ee79d1e9b41" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868"
I20260927 21:39:15.575158 3004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a1d267f0ef4820afdc0ee79d1e9b41" candidate_uuid: "41c064d9af714ad7ad416798f7a14940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef"
I20260927 21:39:15.575385 2725 raft_consensus.cc:3037] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:15.575395 3004 raft_consensus.cc:3037] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:15.577502 2725 raft_consensus.cc:2454] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 1.
I20260927 21:39:15.577611 3004 raft_consensus.cc:2454] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c064d9af714ad7ad416798f7a14940 in term 1.
I20260927 21:39:15.577950 2796 leader_election.cc:302] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03525bbaa2c04165a87dba22117d5868, 41c064d9af714ad7ad416798f7a14940; no voters:
I20260927 21:39:15.578293 3080 raft_consensus.cc:2781] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:15.578480 3080 raft_consensus.cc:686] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 1 LEADER]: Becoming Leader. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Running, Role: LEADER
I20260927 21:39:15.578758 3080 consensus_queue.cc:237] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.581772 2567 catalog_manager.cc:5495] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c064d9af714ad7ad416798f7a14940 (127.0.202.130). New cstate: current_term: 1 leader_uuid: "41c064d9af714ad7ad416798f7a14940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } health_report { overall_health: UNKNOWN } } }
I20260927 21:39:15.607309 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.601546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.0.202.130:33641)
0927 21:39:15.602251 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:15.602256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:15.602273 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:39:15.602333 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:15.602337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:15.602396 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:15.602482 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:15.602957 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.603876 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.605228 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.605681 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.605789 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.606463 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:15.606570 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:15.606642 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:15.606939 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 21:39:15.607004 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":472,"threads_started":1}
I20260927 21:39:15.616953 2725 raft_consensus.cc:1261] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Refusing update from remote peer 41c064d9af714ad7ad416798f7a14940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:15.617841 3004 raft_consensus.cc:1261] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Refusing update from remote peer 41c064d9af714ad7ad416798f7a14940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:15.617820 3080 consensus_queue.cc:1035] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:15.618281 3080 consensus_queue.cc:1035] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:15.661873 2567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:15.662115 2567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:15.665561 2567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Sending DeleteTablet for 3 replicas of tablet 95a1d267f0ef4820afdc0ee79d1e9b41
I20260927 21:39:15.666751 2706 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a1d267f0ef4820afdc0ee79d1e9b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:15 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:33290
I20260927 21:39:15.667115 3158 tablet_replica.cc:330] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: stopping tablet replica
I20260927 21:39:15.667155 2984 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a1d267f0ef4820afdc0ee79d1e9b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:15 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:57958
I20260927 21:39:15.667347 3158 raft_consensus.cc:2229] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:15.667443 3159 tablet_replica.cc:330] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: stopping tablet replica
I20260927 21:39:15.667446 2845 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a1d267f0ef4820afdc0ee79d1e9b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:15 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60782
I20260927 21:39:15.667557 3158 raft_consensus.cc:2258] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:15.667683 3159 raft_consensus.cc:2229] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:15.667747 3157 tablet_replica.cc:330] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: stopping tablet replica
I20260927 21:39:15.667927 3159 raft_consensus.cc:2258] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:15.667979 3157 raft_consensus.cc:2229] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:15.668361 3157 raft_consensus.cc:2258] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:15.668973 3159 ts_tablet_manager.cc:1905] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:15.668973 3158 ts_tablet_manager.cc:1905] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:15.669469 3157 ts_tablet_manager.cc:1905] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:15.672715 2567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:15.673621 2567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:15.675760 3157 ts_tablet_manager.cc:1918] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:15.675976 3157 log.cc:1196] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/95a1d267f0ef4820afdc0ee79d1e9b41
I20260927 21:39:15.676285 3159 ts_tablet_manager.cc:1918] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:15.676538 3157 ts_tablet_manager.cc:1939] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 41c064d9af714ad7ad416798f7a14940: Deleting consensus metadata
I20260927 21:39:15.676545 3158 ts_tablet_manager.cc:1918] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:39:15.676551 3159 log.cc:1196] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/95a1d267f0ef4820afdc0ee79d1e9b41
I20260927 21:39:15.676698 3158 log.cc:1196] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/95a1d267f0ef4820afdc0ee79d1e9b41
I20260927 21:39:15.677160 3159 ts_tablet_manager.cc:1939] T 95a1d267f0ef4820afdc0ee79d1e9b41 P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting consensus metadata
I20260927 21:39:15.677209 3158 ts_tablet_manager.cc:1939] T 95a1d267f0ef4820afdc0ee79d1e9b41 P 03525bbaa2c04165a87dba22117d5868: Deleting consensus metadata
I20260927 21:39:15.677824 2555 catalog_manager.cc:4841] TS 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641): tablet 95a1d267f0ef4820afdc0ee79d1e9b41 (table test-table [id=c57a9a30773143a3ade917ce16d6d1ce]) successfully deleted
I20260927 21:39:15.678329 2556 catalog_manager.cc:4841] TS af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061): tablet 95a1d267f0ef4820afdc0ee79d1e9b41 (table test-table [id=c57a9a30773143a3ade917ce16d6d1ce]) successfully deleted
I20260927 21:39:15.678388 2553 catalog_manager.cc:4841] TS 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761): tablet 95a1d267f0ef4820afdc0ee79d1e9b41 (table test-table [id=c57a9a30773143a3ade917ce16d6d1ce]) successfully deleted
I20260927 21:39:15.681717 2706 tablet_service.cc:1467] Processing CreateTablet for tablet 411f0af4e3844e65843b779e4c2688ed (DEFAULT_TABLE table=test-table [id=a00239672e624df1b117ae4fb1f974ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:15.681897 2845 tablet_service.cc:1467] Processing CreateTablet for tablet 411f0af4e3844e65843b779e4c2688ed (DEFAULT_TABLE table=test-table [id=a00239672e624df1b117ae4fb1f974ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:15.682195 2706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411f0af4e3844e65843b779e4c2688ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:15.682118 2984 tablet_service.cc:1467] Processing CreateTablet for tablet 411f0af4e3844e65843b779e4c2688ed (DEFAULT_TABLE table=test-table [id=a00239672e624df1b117ae4fb1f974ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:15.682386 2845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411f0af4e3844e65843b779e4c2688ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:15.682516 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411f0af4e3844e65843b779e4c2688ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:15.687412 3170 tablet_bootstrap.cc:492] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940: Bootstrap starting.
I20260927 21:39:15.688993 3169 tablet_bootstrap.cc:492] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868: Bootstrap starting.
I20260927 21:39:15.689051 3170 tablet_bootstrap.cc:654] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:15.690522 3169 tablet_bootstrap.cc:654] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:15.690817 3168 tablet_bootstrap.cc:492] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: Bootstrap starting.
I20260927 21:39:15.690946 3170 tablet_bootstrap.cc:492] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940: No bootstrap required, opened a new log
I20260927 21:39:15.691083 3170 ts_tablet_manager.cc:1397] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:39:15.691845 3170 raft_consensus.cc:348] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.692090 3170 raft_consensus.cc:374] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:15.692178 3170 raft_consensus.cc:729] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c064d9af714ad7ad416798f7a14940, State: Initialized, Role: FOLLOWER
I20260927 21:39:15.692412 3168 tablet_bootstrap.cc:654] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:15.692533 3169 tablet_bootstrap.cc:492] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868: No bootstrap required, opened a new log
I20260927 21:39:15.692481 3170 consensus_queue.cc:260] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.692689 3169 ts_tablet_manager.cc:1397] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 21:39:15.693897 3170 ts_tablet_manager.cc:1428] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:15.693782 3169 raft_consensus.cc:348] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.694017 3169 raft_consensus.cc:374] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:15.694097 3169 raft_consensus.cc:729] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03525bbaa2c04165a87dba22117d5868, State: Initialized, Role: FOLLOWER
I20260927 21:39:15.694468 3169 consensus_queue.cc:260] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.694849 3168 tablet_bootstrap.cc:492] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: No bootstrap required, opened a new log
I20260927 21:39:15.695000 3168 ts_tablet_manager.cc:1397] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260927 21:39:15.695115 3169 ts_tablet_manager.cc:1428] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:39:15.695706 3168 raft_consensus.cc:348] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.695871 3168 raft_consensus.cc:374] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:15.695916 3168 raft_consensus.cc:729] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7ddd86f79b4977adb6fe6f8c98a9ef, State: Initialized, Role: FOLLOWER
I20260927 21:39:15.696110 3168 consensus_queue.cc:260] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.696696 3168 ts_tablet_manager.cc:1428] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:39:15.851392 3174 raft_consensus.cc:482] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:15.851673 3174 raft_consensus.cc:504] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.853552 3174 leader_election.cc:288] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641)
I20260927 21:39:15.861940 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.854303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:56433 (local address 127.0.202.130:33641)
0927 21:39:15.854544 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:15.854554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:15.855582 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 21:39:15.855804 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:15.855808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:15.855865 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:15.855962 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:15.857001 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.858124 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.860156 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.861053 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.861236 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.861310 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:15.861572 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:39:15.861697 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:39:15.862413 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.853942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:37753 (local address 127.0.202.129:41761)
0927 21:39:15.855732 (+ 1790us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:15.855739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:15.855763 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:39:15.855839 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:15.855842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:15.855893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:15.856004 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:15.857121 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.858220 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.860092 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.860915 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.861138 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.861227 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:15.861469 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 21:39:15.861594 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1657,"thread_start_us":1503,"threads_started":1}
I20260927 21:39:15.862432 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.853965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.202.131:56433)
0927 21:39:15.855477 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:15.855495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:15.855639 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:15.855975 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:15.855980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:15.856002 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:15.856757 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.856773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.858284 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.858288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.860004 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.860016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.861259 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.861266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.861418 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.861440 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:39:15.861547 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:39:15.861670 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1358,"mutex_wait_us":133,"thread_start_us":777,"threads_started":1}
I20260927 21:39:15.862458 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.853810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.202.131:37753)
0927 21:39:15.855097 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:15.855120 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:15.855675 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:15.856027 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:15.856031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:15.856062 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:15.856863 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.856876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.858326 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.858331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.859933 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.859945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.861139 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.861145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.861315 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.861339 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:39:15.861444 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:39:15.861574 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":5,"thread_start_us":1297,"threads_started":1}
I20260927 21:39:15.863116 2865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411f0af4e3844e65843b779e4c2688ed" candidate_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c064d9af714ad7ad416798f7a14940" is_pre_election: true
I20260927 21:39:15.863411 2865 raft_consensus.cc:2454] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7ddd86f79b4977adb6fe6f8c98a9ef in term 0.
I20260927 21:39:15.863365 2725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411f0af4e3844e65843b779e4c2688ed" candidate_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868" is_pre_election: true
I20260927 21:39:15.863620 2725 raft_consensus.cc:2454] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7ddd86f79b4977adb6fe6f8c98a9ef in term 0.
I20260927 21:39:15.864030 2937 leader_election.cc:302] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c064d9af714ad7ad416798f7a14940, af7ddd86f79b4977adb6fe6f8c98a9ef; no voters:
I20260927 21:39:15.864485 3174 raft_consensus.cc:2781] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:15.864593 3174 raft_consensus.cc:482] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:15.864711 3174 raft_consensus.cc:3037] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:15.866888 3174 raft_consensus.cc:504] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.867581 3174 leader_election.cc:288] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [CANDIDATE]: Term 1 election: Requested vote from peers 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761), 41c064d9af714ad7ad416798f7a14940 (127.0.202.130:33641)
I20260927 21:39:15.868090 2725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411f0af4e3844e65843b779e4c2688ed" candidate_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03525bbaa2c04165a87dba22117d5868"
I20260927 21:39:15.868090 2865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411f0af4e3844e65843b779e4c2688ed" candidate_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c064d9af714ad7ad416798f7a14940"
I20260927 21:39:15.868357 2725 raft_consensus.cc:3037] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:15.868359 2865 raft_consensus.cc:3037] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:15.871304 2725 raft_consensus.cc:2454] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7ddd86f79b4977adb6fe6f8c98a9ef in term 1.
I20260927 21:39:15.871304 2865 raft_consensus.cc:2454] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7ddd86f79b4977adb6fe6f8c98a9ef in term 1.
I20260927 21:39:15.871737 2937 leader_election.cc:302] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03525bbaa2c04165a87dba22117d5868, 41c064d9af714ad7ad416798f7a14940, af7ddd86f79b4977adb6fe6f8c98a9ef; no voters:
I20260927 21:39:15.872107 3174 raft_consensus.cc:2781] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:15.872664 3174 raft_consensus.cc:686] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 LEADER]: Becoming Leader. State: Replica: af7ddd86f79b4977adb6fe6f8c98a9ef, State: Running, Role: LEADER
I20260927 21:39:15.873080 3174 consensus_queue.cc:237] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } }
I20260927 21:39:15.876233 2569 catalog_manager.cc:5495] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131). New cstate: current_term: 1 leader_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7ddd86f79b4977adb6fe6f8c98a9ef" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42061 } health_report { overall_health: HEALTHY } } }
I20260927 21:39:15.910511 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.903840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.0.202.131:42061)
0927 21:39:15.904543 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:15.904548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:15.904566 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:39:15.904651 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:15.904654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:15.904724 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:15.904941 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:15.905340 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.906204 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.907591 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.908188 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.908367 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.909074 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:15.909790 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:15.909908 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:15.910026 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:39:15.910093 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":446,"threads_started":1}
I20260927 21:39:15.927393 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.921383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:40053 (local address 127.0.202.130:33641)
0927 21:39:15.921585 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:15.921593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:15.922206 (+ 613us) server_negotiation.cc:407] Connection header received
0927 21:39:15.922347 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:15.922351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:15.922415 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:15.922488 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:15.922994 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.923746 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.925667 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.926200 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.926395 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.926819 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:15.926924 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:15.926995 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:15.927099 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 21:39:15.927164 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:39:15.927469 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.921383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.202.131:40053)
0927 21:39:15.922088 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:15.922105 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:15.922240 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:15.922519 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:15.922522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:15.922532 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:15.922862 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.922870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.923902 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.923906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.925517 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.925530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.926397 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.926402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.926523 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.926579 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:15.927014 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:15.927022 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:39:15.927084 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:39:15.927199 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":414,"threads_started":1}
I20260927 21:39:15.937027 2865 raft_consensus.cc:1261] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Refusing update from remote peer af7ddd86f79b4977adb6fe6f8c98a9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:15.937034 2725 raft_consensus.cc:1261] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Refusing update from remote peer af7ddd86f79b4977adb6fe6f8c98a9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:15.937814 3188 consensus_queue.cc:1035] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "03525bbaa2c04165a87dba22117d5868" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:15.938258 3174 consensus_queue.cc:1035] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c064d9af714ad7ad416798f7a14940" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:15.965791 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.959966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:53957 (local address 127.0.202.131:42061)
0927 21:39:15.960240 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:15.960249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:15.960735 (+ 486us) server_negotiation.cc:407] Connection header received
0927 21:39:15.960856 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:15.960859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:15.960920 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:15.960991 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:15.961382 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.962371 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.964013 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.964545 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.964698 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.965091 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:15.965204 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:15.965299 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:15.965451 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:39:15.965523 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 21:39:15.965956 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:15.960002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42061 (local address 127.0.202.130:53957)
0927 21:39:15.960635 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:15.960651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:15.960763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:15.960995 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:15.960997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:15.961004 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:39:15.961253 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.961260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.962498 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.962506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.963872 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:15.963885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:15.964742 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:15.964751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:15.964893 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:15.964933 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:39:15.965299 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:39:15.965334 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:39:15.965431 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:39:15.965561 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":367,"threads_started":1}
W20260927 21:39:15.967095 3140 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:39:17.568668 2569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:17.568898 2569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:17.572935 2569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ccad99ce80405c9ebb576c5ee231e9: Sending DeleteTablet for 3 replicas of tablet 411f0af4e3844e65843b779e4c2688ed
I20260927 21:39:17.573793 2706 tablet_service.cc:1514] Processing DeleteTablet for tablet 411f0af4e3844e65843b779e4c2688ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:17 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:33290
I20260927 21:39:17.573920 2845 tablet_service.cc:1514] Processing DeleteTablet for tablet 411f0af4e3844e65843b779e4c2688ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:17 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60782
I20260927 21:39:17.574019 2984 tablet_service.cc:1514] Processing DeleteTablet for tablet 411f0af4e3844e65843b779e4c2688ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:17 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:57958
I20260927 21:39:17.574666 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2640
I20260927 21:39:17.588315 3224 tablet_replica.cc:330] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868: stopping tablet replica
I20260927 21:39:17.587782 3225 tablet_replica.cc:330] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940: stopping tablet replica
I20260927 21:39:17.588624 3226 tablet_replica.cc:330] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: stopping tablet replica
I20260927 21:39:17.588629 3224 raft_consensus.cc:2229] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:17.588862 3226 raft_consensus.cc:2229] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:17.588840 3225 raft_consensus.cc:2229] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:17.588945 3224 raft_consensus.cc:2258] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:17.589165 3225 raft_consensus.cc:2258] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:17.589984 3224 ts_tablet_manager.cc:1905] T 411f0af4e3844e65843b779e4c2688ed P 03525bbaa2c04165a87dba22117d5868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:17.589977 3226 raft_consensus.cc:2258] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:17.590279 3225 ts_tablet_manager.cc:1905] T 411f0af4e3844e65843b779e4c2688ed P 41c064d9af714ad7ad416798f7a14940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:17.590981 3226 ts_tablet_manager.cc:1905] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:39:17 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:17.614297 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.576001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.202.129:41761)
0927 21:39:17.585984 (+ 9983us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:17.585993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:17.586037 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:39:17.586121 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:17.586129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:17.586236 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:17.586389 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:17.588763 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:17.590328 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:17.592246 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:17.592946 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:17.593065 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:17.607595 (+ 14530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:17.607620 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:17.607636 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:17.607685 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:17.610792 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:17.611619 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:17.611628 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:17.611631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:17.611885 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:17.612420 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:17.612424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:17.612426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:17.612941 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:17.613159 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:17.613578 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 21:39:17.613927 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9795,"thread_start_us":9622,"threads_started":1}
W20260927 21:39:17.615008 2771 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33316
W20260927 21:39:17.615681 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2640: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:17.615928 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 2640
I20260927 21:39:17.658155 3226 ts_tablet_manager.cc:1918] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:39:17.658329 3226 log.cc:1196] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/411f0af4e3844e65843b779e4c2688ed
I20260927 21:39:17.658762 3226 ts_tablet_manager.cc:1939] T 411f0af4e3844e65843b779e4c2688ed P af7ddd86f79b4977adb6fe6f8c98a9ef: Deleting consensus metadata
W20260927 21:39:17.677074 2553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.202.129:41761 (error 108)
I20260927 21:39:17.677980 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2780
I20260927 21:39:17.679445 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.678544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.0.1:33326)
0927 21:39:17.679047 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.679209 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":606,"threads_started":1}
W20260927 21:39:17.679633 2553 catalog_manager.cc:4584] TS 03525bbaa2c04165a87dba22117d5868 (127.0.202.129:41761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 411f0af4e3844e65843b779e4c2688ed: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41761: connect: Connection refused (error 111)
Sep 27 21:39:17 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:39:17.694504 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.678799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.0.202.130:33641)
0927 21:39:17.679313 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:17.679317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:17.679334 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:39:17.679386 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:17.679397 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:17.679453 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:17.679644 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:17.681079 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:17.681826 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:17.683095 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:17.683550 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:17.683690 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:17.688389 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:17.688405 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:17.688421 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:17.688456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:17.691135 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:17.691887 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:17.691903 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:17.691905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:17.692029 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:17.692849 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:17.692857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:17.692861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:17.693241 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:17.693595 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:17.693896 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:39:17.694145 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":269,"threads_started":1}
W20260927 21:39:17.695214 2910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60800
W20260927 21:39:17.696162 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2780: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:17.696380 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 2780
I20260927 21:39:17.731557 2556 catalog_manager.cc:4841] TS af7ddd86f79b4977adb6fe6f8c98a9ef (127.0.202.131:42061): tablet 411f0af4e3844e65843b779e4c2688ed (table test-table [id=a00239672e624df1b117ae4fb1f974ae]) successfully deleted
I20260927 21:39:17.733315 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.732861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.0.1:33342)
0927 21:39:17.733080 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.733165 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 21:39:17.734562 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.734108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.0.1:33346)
0927 21:39:17.734299 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.734408 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:39:17.784469 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2919
I20260927 21:39:17.784746 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.784267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.0.1:60810)
0927 21:39:17.784472 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.784589 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
Sep 27 21:39:17 dist-test-slave-25dq krb5kdc[2524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:39:17.795405 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.794895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.0.1:33360)
0927 21:39:17.795166 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.795245 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 21:39:17.796885 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.796454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41761 (local address 127.0.0.1:33374)
0927 21:39:17.796639 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.796730 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:39:17.803308 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.785257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.0.202.131:42061)
0927 21:39:17.785909 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:17.785914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:17.785930 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:39:17.785983 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:17.785987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:17.786051 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:17.786163 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:17.788109 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:17.789087 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:17.790871 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:17.791295 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:17.791405 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:17.796446 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:17.796462 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:17.796480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:17.796553 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:17.800230 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:17.801190 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:17.801199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:17.801220 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:17.801396 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:17.801767 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:17.801770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:17.801773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:17.802185 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:17.802415 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:17.802718 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:39:17.802923 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":308,"threads_started":1}
W20260927 21:39:17.804055 3049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57974
W20260927 21:39:17.804730 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2919: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:17.804879 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 2919
I20260927 21:39:17.817636 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.817155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.0.1:60820)
0927 21:39:17.817367 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.817464 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 21:39:17.818715 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.818334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.0.1:60832)
0927 21:39:17.818530 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.818601 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:39:17.828550 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 2537
I20260927 21:39:17.829723 2609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:17.946244 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:17.945490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33641 (local address 127.0.0.1:60834)
0927 21:39:17.945776 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:17.945919 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.130:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 21:39:17.946448 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 2537
2026-09-27T21:39:17Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8726 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:20 dist-test-slave-25dq krb5kdc[3237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:20 dist-test-slave-25dq krb5kdc[3237](info): set up 2 sockets
Sep 27 21:39:20 dist-test-slave-25dq krb5kdc[3237](info): commencing operation
krb5kdc: starting...
W20260927 21:39:21.105129 810 mini_kdc.cc:121] Time spent starting KDC: real 3.111s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 21:39:21.535684 810 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.430s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 21:39:21.683141 810 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.147s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:21 dist-test-slave-25dq krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:21.829115 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:34073
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:34073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:22.162676 3250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:22.163053 3250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:22.163226 3250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:22.170594 3250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:22.170774 3250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:22.170890 3250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:22.171000 3250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:22.171100 3250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:22.180940 3250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:34073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:34073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:22.182332 3250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:22.183679 3250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:22.196817 3255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:22.196887 3256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:22.197005 3258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:22.197655 3250 server_base.cc:1034] running on GCE node
I20260927 21:39:22.198573 3250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:22.200212 3250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:22.201505 3250 hybrid_clock.cc:648] HybridClock initialized: now 1790545162201435 us; error 62 us; skew 500 ppm
I20260927 21:39:22.201907 3250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:22 dist-test-slave-25dq krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:22.205291 3250 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:22.206933 3250 webserver.cc:460] Webserver started at http://127.0.202.190:46547/ using document root <none> and password file <none>
I20260927 21:39:22.207655 3250 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:22.207800 3250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:22.208109 3250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:22.211138 3250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "2538798a76794b4b94478be087ac2af9"
format_stamp: "Formatted at 2026-09-27 21:39:22 on dist-test-slave-25dq"
server_key: "6fbdf8aef2181601486b01bcc84c5a9d"
server_key_iv: "f6a7cc4f22f0c0df676ca17db74153d0"
server_key_version: "encryptionkey@0"
I20260927 21:39:22.211923 3250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "2538798a76794b4b94478be087ac2af9"
format_stamp: "Formatted at 2026-09-27 21:39:22 on dist-test-slave-25dq"
server_key: "6fbdf8aef2181601486b01bcc84c5a9d"
server_key_iv: "f6a7cc4f22f0c0df676ca17db74153d0"
server_key_version: "encryptionkey@0"
I20260927 21:39:22.217187 3250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:39:22.221036 3265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:22.222568 3250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:22.222757 3250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "2538798a76794b4b94478be087ac2af9"
format_stamp: "Formatted at 2026-09-27 21:39:22 on dist-test-slave-25dq"
server_key: "6fbdf8aef2181601486b01bcc84c5a9d"
server_key_iv: "f6a7cc4f22f0c0df676ca17db74153d0"
server_key_version: "encryptionkey@0"
I20260927 21:39:22.222961 3250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:22.257982 3250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:22.262738 3250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:22.263047 3250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:22.282351 3250 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:34073
I20260927 21:39:22.282388 3316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:34073 every 8 connection(s)
I20260927 21:39:22.283782 3250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:22.287786 3317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:22.289300 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 3250
I20260927 21:39:22.289573 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:22.290462 810 external_mini_cluster.cc:1428] Setting key 4597d284d8323c2b62412b96e26670b7
I20260927 21:39:22.301767 3317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9: Bootstrap starting.
I20260927 21:39:22.305233 3317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9: Neither blocks nor log segments found. Creating new log.
Sep 27 21:39:22 dist-test-slave-25dq krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:22.306610 3317 log.cc:824] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:22.310567 3317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9: No bootstrap required, opened a new log
I20260927 21:39:22.317436 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:22.292853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.0.202.190:34073)
0927 21:39:22.295380 (+ 2527us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:22.295394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:22.295439 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:39:22.297297 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:22.297322 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:22.297737 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:22.298133 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:22.299655 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:22.301051 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:22.302408 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:22.303095 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:22.303317 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:22.308081 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:22.308094 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:22.308118 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:22.308178 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:22.312473 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:22.313219 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:22.313228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:22.313238 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:22.313468 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:22.313828 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:22.313835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:22.313844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:22.314644 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:22.314959 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:22.315365 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 21:39:22.315885 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2322,"thread_start_us":2093,"threads_started":1}
I20260927 21:39:22.322118 3317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538798a76794b4b94478be087ac2af9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34073 } }
I20260927 21:39:22.322484 3317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:22.322566 3317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2538798a76794b4b94478be087ac2af9, State: Initialized, Role: FOLLOWER
I20260927 21:39:22.323104 3317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538798a76794b4b94478be087ac2af9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34073 } }
I20260927 21:39:22.323323 3317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:22.323423 3317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:22.323557 3317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:22.326263 3317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538798a76794b4b94478be087ac2af9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34073 } }
I20260927 21:39:22.326781 3317 leader_election.cc:302] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2538798a76794b4b94478be087ac2af9; no voters:
I20260927 21:39:22.327404 3317 leader_election.cc:288] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:22.327567 3322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:22.328337 3322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [term 1 LEADER]: Becoming Leader. State: Replica: 2538798a76794b4b94478be087ac2af9, State: Running, Role: LEADER
I20260927 21:39:22.328884 3322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538798a76794b4b94478be087ac2af9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34073 } }
I20260927 21:39:22.329782 3317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:22.333191 3323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2538798a76794b4b94478be087ac2af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538798a76794b4b94478be087ac2af9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34073 } } }
I20260927 21:39:22.333600 3323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:22.334185 3324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2538798a76794b4b94478be087ac2af9. Latest consensus state: current_term: 1 leader_uuid: "2538798a76794b4b94478be087ac2af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538798a76794b4b94478be087ac2af9" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34073 } } }
I20260927 21:39:22.334457 3324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:22.335343 3330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:22.341027 3330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:22.353404 3330 catalog_manager.cc:1347] Generated new cluster ID: 46c45f8f095242f381efe186edc3ece5
I20260927 21:39:22.353600 3330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:22.374696 3330 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:22.376066 3330 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:22.384131 3330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2538798a76794b4b94478be087ac2af9: Generated new TSK 0
I20260927 21:39:22.385073 3330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:22.445035 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34073
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:22.805943 3343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:22.806229 3343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:22.806396 3343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:22.813794 3343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:22.813911 3343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:22.814086 3343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:22.824515 3343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:22.825918 3343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:22.827338 3343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:22.841653 3349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:22.841653 3348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:22.841655 3351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:22.842593 3343 server_base.cc:1034] running on GCE node
I20260927 21:39:22.843263 3343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:22.844405 3343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:22.845604 3343 hybrid_clock.cc:648] HybridClock initialized: now 1790545162845556 us; error 35 us; skew 500 ppm
I20260927 21:39:22.845965 3343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:22 dist-test-slave-25dq krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:22.854311 3343 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:22.856205 3343 webserver.cc:460] Webserver started at http://127.0.202.129:32849/ using document root <none> and password file <none>
I20260927 21:39:22.857028 3343 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:22.857189 3343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:22.857529 3343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:22.861290 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "7307b67f7fbd4f8993976922edb03cbb"
format_stamp: "Formatted at 2026-09-27 21:39:22 on dist-test-slave-25dq"
server_key: "f9d700b5e4377df184dc6893eeae581d"
server_key_iv: "222b864761d944021573e1bae9437241"
server_key_version: "encryptionkey@0"
I20260927 21:39:22.862161 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "7307b67f7fbd4f8993976922edb03cbb"
format_stamp: "Formatted at 2026-09-27 21:39:22 on dist-test-slave-25dq"
server_key: "f9d700b5e4377df184dc6893eeae581d"
server_key_iv: "222b864761d944021573e1bae9437241"
server_key_version: "encryptionkey@0"
I20260927 21:39:22.867933 3343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 21:39:22.871876 3358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:22.873407 3343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:22.873588 3343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "7307b67f7fbd4f8993976922edb03cbb"
format_stamp: "Formatted at 2026-09-27 21:39:22 on dist-test-slave-25dq"
server_key: "f9d700b5e4377df184dc6893eeae581d"
server_key_iv: "222b864761d944021573e1bae9437241"
server_key_version: "encryptionkey@0"
I20260927 21:39:22.873796 3343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:22.889779 3343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:22.894358 3343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:22.894680 3343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:22.895694 3343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:22.897145 3343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:22.897245 3343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:22.897394 3343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:22.897468 3343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:22.932958 3343 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:40449
I20260927 21:39:22.933049 3470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:40449 every 8 connection(s)
I20260927 21:39:22.934332 3343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:22.937776 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 3343
I20260927 21:39:22.938019 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:22.938870 810 external_mini_cluster.cc:1428] Setting key d3fd2a9fce1d57dbaef642b9c4847237
Sep 27 21:39:22 dist-test-slave-25dq krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:22.958644 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:22.937436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:36849 (local address 127.0.202.190:34073)
0927 21:39:22.938108 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:22.938117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:22.939243 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 21:39:22.941789 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:22.941793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:22.941858 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:22.941951 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:22.944493 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:22.945253 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:22.947908 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:22.948452 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:22.948599 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:22.952434 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:22.952439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:22.952443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:22.952482 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:22.955271 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:22.956102 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:22.956108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:22.956112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:22.956251 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:22.956779 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:22.956787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:22.956790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:22.957089 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:22.957248 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:22.958031 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 21:39:22.958265 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":220,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:39:22.959723 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:22.938095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:34073 (local address 127.0.202.129:36849)
0927 21:39:22.939035 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:22.939102 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:22.941436 (+ 2334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:22.942270 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:22.942287 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:22.943236 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:22.944202 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:22.944234 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:22.945438 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:22.945447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:22.947735 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:22.947749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:22.948713 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:22.948721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:22.948974 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:22.949742 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:22.949765 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:22.952137 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:22.955454 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:22.955464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:22.955492 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:22.955956 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:22.956440 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:22.956445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:22.956450 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:22.956657 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:22.957237 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:22.957246 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:22.957634 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 21:39:22.957919 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 21:39:22.958520 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":312,"threads_started":1}
I20260927 21:39:22.961623 3471 heartbeater.cc:347] Connected to a master server at 127.0.202.190:34073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:39:22.962138 3471 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:22.963148 3471 heartbeater.cc:511] Master 127.0.202.190:34073 requested a full tablet report, sending...
I20260927 21:39:22.965796 3282 ts_manager.cc:194] Registered new tserver with Master: 7307b67f7fbd4f8993976922edb03cbb (127.0.202.129:40449)
I20260927 21:39:22.967478 3282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:36849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:22.989529 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34073
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:23.327521 3477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:23.327822 3477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:23.327924 3477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:23.335381 3477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:23.335500 3477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:23.335614 3477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:23.345685 3477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:23.346985 3477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:23.348326 3477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:23.362176 3482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:23.362187 3483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:23.362375 3485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:23.363081 3477 server_base.cc:1034] running on GCE node
I20260927 21:39:23.363678 3477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:23.364687 3477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:23.365895 3477 hybrid_clock.cc:648] HybridClock initialized: now 1790545163365782 us; error 102 us; skew 500 ppm
I20260927 21:39:23.366206 3477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:23 dist-test-slave-25dq krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:23.369673 3477 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:23.371412 3477 webserver.cc:460] Webserver started at http://127.0.202.130:39965/ using document root <none> and password file <none>
I20260927 21:39:23.372521 3477 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:23.372637 3477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:23.373025 3477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:23.376029 3477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "22c0aa2bc59c4e9d811f409e02646208"
format_stamp: "Formatted at 2026-09-27 21:39:23 on dist-test-slave-25dq"
server_key: "1818c0680b611b4955ac9f6c136d5433"
server_key_iv: "b58ace03461dc497fd3051174c356375"
server_key_version: "encryptionkey@0"
I20260927 21:39:23.376912 3477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "22c0aa2bc59c4e9d811f409e02646208"
format_stamp: "Formatted at 2026-09-27 21:39:23 on dist-test-slave-25dq"
server_key: "1818c0680b611b4955ac9f6c136d5433"
server_key_iv: "b58ace03461dc497fd3051174c356375"
server_key_version: "encryptionkey@0"
I20260927 21:39:23.383018 3477 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:39:23.386812 3492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:23.388477 3477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:39:23.388660 3477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "22c0aa2bc59c4e9d811f409e02646208"
format_stamp: "Formatted at 2026-09-27 21:39:23 on dist-test-slave-25dq"
server_key: "1818c0680b611b4955ac9f6c136d5433"
server_key_iv: "b58ace03461dc497fd3051174c356375"
server_key_version: "encryptionkey@0"
I20260927 21:39:23.388870 3477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:23.405413 3477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:23.409895 3477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:23.410202 3477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:23.411224 3477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:23.412774 3477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:23.412897 3477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:23.413069 3477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:23.413144 3477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:23.447456 3477 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:38611
I20260927 21:39:23.447549 3604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:38611 every 8 connection(s)
I20260927 21:39:23.448818 3477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:23.449877 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 3477
I20260927 21:39:23.450169 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:23.450721 810 external_mini_cluster.cc:1428] Setting key 3232ea42214b31637f86b54639477e19
Sep 27 21:39:23 dist-test-slave-25dq krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:39:23.471479 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:23.451753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:40403 (local address 127.0.202.190:34073)
0927 21:39:23.451958 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:23.451964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:23.453343 (+ 1379us) server_negotiation.cc:407] Connection header received
0927 21:39:23.455148 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:23.455154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:23.455240 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:23.455358 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:23.456958 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.458104 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.459664 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.460516 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.460665 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:23.464458 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:23.464467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:23.464475 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:23.464524 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:23.468217 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:23.469064 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:23.469072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:23.469076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:23.469236 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:23.469726 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:23.469731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:23.469735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:23.470010 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:23.470185 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:23.470881 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 21:39:23.471111 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:39:23.472661 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:23.452248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:34073 (local address 127.0.202.130:40403)
0927 21:39:23.453182 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:23.453230 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:23.454870 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:23.455603 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:23.455612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:23.456120 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:23.456773 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:23.456790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.458279 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.458284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:23.459509 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:23.459522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.460762 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.460767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:23.460978 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:23.461685 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:23.461708 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:23.464170 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:23.468434 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:23.468442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:23.468458 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:23.468915 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:23.469349 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:23.469353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:23.469355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:23.469597 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:23.470156 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:23.470164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:23.470533 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 21:39:23.470805 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 21:39:23.471359 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":345,"threads_started":1}
I20260927 21:39:23.474287 3605 heartbeater.cc:347] Connected to a master server at 127.0.202.190:34073
I20260927 21:39:23.474622 3605 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:23.475544 3605 heartbeater.cc:511] Master 127.0.202.190:34073 requested a full tablet report, sending...
I20260927 21:39:23.477283 3282 ts_manager.cc:194] Registered new tserver with Master: 22c0aa2bc59c4e9d811f409e02646208 (127.0.202.130:38611)
I20260927 21:39:23.478256 3282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:40403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:23.496723 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34073
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:23.839366 3610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:23.839699 3610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:23.839864 3610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:23.847311 3610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:23.847483 3610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:23.847656 3610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:23.857851 3610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:23.859263 3610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:23.860747 3610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:23.874953 3615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:23.875336 3618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:23.876140 3616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:23.877120 3610 server_base.cc:1034] running on GCE node
I20260927 21:39:23.877848 3610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:23.878975 3610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:23.880198 3610 hybrid_clock.cc:648] HybridClock initialized: now 1790545163880121 us; error 64 us; skew 500 ppm
I20260927 21:39:23.880688 3610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:23 dist-test-slave-25dq krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:23.884217 3610 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:23.885951 3610 webserver.cc:460] Webserver started at http://127.0.202.131:37711/ using document root <none> and password file <none>
I20260927 21:39:23.886729 3610 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:23.886885 3610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:23.887223 3610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:23.890552 3610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "13a900834e4249808eb4cd8bcc8728d6"
format_stamp: "Formatted at 2026-09-27 21:39:23 on dist-test-slave-25dq"
server_key: "de3b9c93e04baf77d6cf5bf201aee772"
server_key_iv: "81e985ffeca097c1b0de0783f6700320"
server_key_version: "encryptionkey@0"
I20260927 21:39:23.891391 3610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "13a900834e4249808eb4cd8bcc8728d6"
format_stamp: "Formatted at 2026-09-27 21:39:23 on dist-test-slave-25dq"
server_key: "de3b9c93e04baf77d6cf5bf201aee772"
server_key_iv: "81e985ffeca097c1b0de0783f6700320"
server_key_version: "encryptionkey@0"
I20260927 21:39:23.897521 3610 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:39:23.901274 3625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:23.902719 3610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:23.902920 3610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "13a900834e4249808eb4cd8bcc8728d6"
format_stamp: "Formatted at 2026-09-27 21:39:23 on dist-test-slave-25dq"
server_key: "de3b9c93e04baf77d6cf5bf201aee772"
server_key_iv: "81e985ffeca097c1b0de0783f6700320"
server_key_version: "encryptionkey@0"
I20260927 21:39:23.903194 3610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:23.919705 3610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:23.924748 3610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:23.925064 3610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:23.926189 3610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:23.927634 3610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:23.927739 3610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:23.927888 3610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:23.927971 3610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:23.969687 3610 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37815
I20260927 21:39:23.969733 3737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37815 every 8 connection(s)
I20260927 21:39:23.970743 3471 heartbeater.cc:503] Master 127.0.202.190:34073 was elected leader, sending a full tablet report...
I20260927 21:39:23.971170 3610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:23.977103 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 3610
I20260927 21:39:23.977319 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:23.977736 810 external_mini_cluster.cc:1428] Setting key f411b6b9ca61855dfce571d82b84cd58
Sep 27 21:39:23 dist-test-slave-25dq krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:23.993528 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:23.974007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:39719 (local address 127.0.202.190:34073)
0927 21:39:23.974480 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:23.974485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:23.975569 (+ 1084us) server_negotiation.cc:407] Connection header received
0927 21:39:23.977886 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:23.977891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:23.977994 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:23.978121 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:23.979971 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.980913 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.982632 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.983071 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.983218 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:23.987293 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:23.987302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:23.987305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:23.987340 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:23.990135 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:23.990943 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:23.990950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:23.990953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:23.991103 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:23.991600 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:23.991604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:23.991606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:23.991875 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:23.992099 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:23.992854 (+ 755us) server_negotiation.cc:299] Negotiation successful
0927 21:39:23.993084 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":258,"threads_started":1}
I20260927 21:39:23.994563 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:23.974639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:34073 (local address 127.0.202.131:39719)
0927 21:39:23.975400 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:23.975445 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:23.977593 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:23.978411 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:23.978420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:23.978991 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:23.979776 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:23.979798 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.981085 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.981096 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:23.982474 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:23.982486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:23.983314 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:23.983321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:23.983598 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:23.984262 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:23.984285 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:23.986928 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:23.990296 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:23.990304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:23.990318 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:23.990788 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:23.991231 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:23.991236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:23.991239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:23.991440 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:23.992009 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:23.992026 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:23.992483 (+ 457us) client_negotiation.cc:769] Sending connection context
0927 21:39:23.992766 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 21:39:23.993343 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":330,"threads_started":1}
I20260927 21:39:23.996150 3738 heartbeater.cc:347] Connected to a master server at 127.0.202.190:34073
I20260927 21:39:23.996639 3738 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:23.997409 3738 heartbeater.cc:511] Master 127.0.202.190:34073 requested a full tablet report, sending...
I20260927 21:39:23.998855 3282 ts_manager.cc:194] Registered new tserver with Master: 13a900834e4249808eb4cd8bcc8728d6 (127.0.202.131:37815)
I20260927 21:39:23.999747 3282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:39719
I20260927 21:39:24.006058 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:39:24.038843 3740 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:24.035957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.202.190:34073)
0927 21:39:24.036226 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:24.036236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:24.036788 (+ 552us) server_negotiation.cc:407] Connection header received
0927 21:39:24.037024 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:24.037030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:24.037119 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:24.037231 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:24.038638 (+ 1407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51604: BlockingRecv error: recv got EOF from 127.0.0.1:51604 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 21:39:24.042459 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3343
W20260927 21:39:24.045540 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3343: Not authorized: Client connection negotiation failed: client connection to 127.0.202.129:40449: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:39:24.045753 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 3343
I20260927 21:39:24.061308 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3477
W20260927 21:39:24.064138 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3477: Not authorized: Client connection negotiation failed: client connection to 127.0.202.130:38611: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:39:24.064340 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 3477
I20260927 21:39:24.080565 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3610
W20260927 21:39:24.083814 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3610: Not authorized: Client connection negotiation failed: client connection to 127.0.202.131:37815: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:39:24.083963 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 3610
I20260927 21:39:24.096541 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3250
I20260927 21:39:24.097883 3312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:24.199839 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 3250
2026-09-27T21:39:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (6250 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:24 dist-test-slave-25dq krb5kdc[3756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:24 dist-test-slave-25dq krb5kdc[3756](info): set up 2 sockets
Sep 27 21:39:24 dist-test-slave-25dq krb5kdc[3756](info): commencing operation
krb5kdc: starting...
W20260927 21:39:25.308987 810 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:25 dist-test-slave-25dq krb5kdc[3756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:25.517666 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:34603
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:40451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:34603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:25.861801 3769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:25.862192 3769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:25.862356 3769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:25.869867 3769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:25.870061 3769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:25.870146 3769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:25.870254 3769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:25.870337 3769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:25.880692 3769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:40451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:34603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:34603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:25.882143 3769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:25.883538 3769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:25.895829 3777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:25.895803 3774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:25.895859 3775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:25.896544 3769 server_base.cc:1034] running on GCE node
I20260927 21:39:25.897471 3769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:25.899058 3769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:25.900435 3769 hybrid_clock.cc:648] HybridClock initialized: now 1790545165900371 us; error 44 us; skew 500 ppm
I20260927 21:39:25.900966 3769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:25 dist-test-slave-25dq krb5kdc[3756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:25.904585 3769 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:25.906261 3769 webserver.cc:460] Webserver started at http://127.0.202.190:36239/ using document root <none> and password file <none>
I20260927 21:39:25.907015 3769 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:25.907167 3769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:25.907483 3769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:25.910683 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "0ff6a201fb18418c8caf8fcda8446b69"
format_stamp: "Formatted at 2026-09-27 21:39:25 on dist-test-slave-25dq"
server_key: "dcce61429f67790f0cfadcb85e2aaca3"
server_key_iv: "6a73e379d108c4c365889c755dea3dc0"
server_key_version: "encryptionkey@0"
I20260927 21:39:25.911526 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "0ff6a201fb18418c8caf8fcda8446b69"
format_stamp: "Formatted at 2026-09-27 21:39:25 on dist-test-slave-25dq"
server_key: "dcce61429f67790f0cfadcb85e2aaca3"
server_key_iv: "6a73e379d108c4c365889c755dea3dc0"
server_key_version: "encryptionkey@0"
I20260927 21:39:25.917378 3769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:39:25.921296 3784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:25.922838 3769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:25.923048 3769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "0ff6a201fb18418c8caf8fcda8446b69"
format_stamp: "Formatted at 2026-09-27 21:39:25 on dist-test-slave-25dq"
server_key: "dcce61429f67790f0cfadcb85e2aaca3"
server_key_iv: "6a73e379d108c4c365889c755dea3dc0"
server_key_version: "encryptionkey@0"
I20260927 21:39:25.923269 3769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:25.940318 3769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:25.944753 3769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:25.945043 3769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:25.964359 3769 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:34603
I20260927 21:39:25.964468 3835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:34603 every 8 connection(s)
I20260927 21:39:25.965796 3769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:25.968662 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 3769
I20260927 21:39:25.968883 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:25.969717 810 external_mini_cluster.cc:1428] Setting key f6e44b68b54d532526d0f69274008689
I20260927 21:39:25.970607 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:39:25 dist-test-slave-25dq krb5kdc[3756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:25.984053 3836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69: Bootstrap starting.
I20260927 21:39:25.988018 3836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:25.989162 3836 log.cc:824] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:25.992127 3836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69: No bootstrap required, opened a new log
I20260927 21:39:25.993013 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:25.971767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.0.202.190:34603)
0927 21:39:25.972621 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:25.972634 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:25.972677 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:39:25.974262 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:25.974285 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:25.974696 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:25.975132 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:25.976689 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:25.978028 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:25.979278 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:25.979967 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:25.980184 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:25.984352 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:25.984365 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:25.984388 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:25.984426 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:25.988208 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:25.989062 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:25.989072 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:25.989082 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:25.989371 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:25.989835 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:25.989841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:25.989845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:25.990475 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:25.990836 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:25.991339 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 21:39:25.991805 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":407,"threads_started":1}
I20260927 21:39:25.997323 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:39:26.000617 3836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff6a201fb18418c8caf8fcda8446b69" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34603 } }
I20260927 21:39:26.001065 3836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:26.001174 3836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff6a201fb18418c8caf8fcda8446b69, State: Initialized, Role: FOLLOWER
I20260927 21:39:26.001762 3836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff6a201fb18418c8caf8fcda8446b69" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34603 } }
I20260927 21:39:26.002009 3836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:26.002157 3836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:26.002328 3836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:26.005617 3836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff6a201fb18418c8caf8fcda8446b69" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34603 } }
I20260927 21:39:26.006232 3836 leader_election.cc:302] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ff6a201fb18418c8caf8fcda8446b69; no voters:
I20260927 21:39:26.006982 3836 leader_election.cc:288] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:26.007157 3842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:26.008435 3842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [term 1 LEADER]: Becoming Leader. State: Replica: 0ff6a201fb18418c8caf8fcda8446b69, State: Running, Role: LEADER
I20260927 21:39:26.008967 3842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff6a201fb18418c8caf8fcda8446b69" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34603 } }
I20260927 21:39:26.009356 3836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:26.011811 3844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ff6a201fb18418c8caf8fcda8446b69. Latest consensus state: current_term: 1 leader_uuid: "0ff6a201fb18418c8caf8fcda8446b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff6a201fb18418c8caf8fcda8446b69" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34603 } } }
I20260927 21:39:26.012639 3843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ff6a201fb18418c8caf8fcda8446b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff6a201fb18418c8caf8fcda8446b69" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34603 } } }
I20260927 21:39:26.012982 3843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:26.013402 3844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:26.015151 3848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:26.020921 3848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:26.032934 3848 catalog_manager.cc:1347] Generated new cluster ID: 33f4dad8fa764191b11eb8c4925e3ea6
I20260927 21:39:26.033150 3848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:26.054282 3848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:26.056912 3848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:26.058389 3866 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:39:26.067209 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:26.056359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.0.202.190:34603)
0927 21:39:26.056545 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:26.056554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:26.057727 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 21:39:26.057929 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:26.057938 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:26.058026 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:26.058155 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:26.059818 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:26.061055 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:26.063033 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:26.063809 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:26.063966 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:26.065268 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:26.065273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:26.065278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:26.066261 (+ 983us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:26.066319 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:26.066344 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:26.066503 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:26.066614 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:39:26.066739 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":59}
I20260927 21:39:26.067677 3848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ff6a201fb18418c8caf8fcda8446b69: Generated new TSK 0
I20260927 21:39:26.068845 3848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:39:26.077971 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3769
I20260927 21:39:26.079607 3831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:26.167135 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 3769
2026-09-27T21:39:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1959 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:26 dist-test-slave-25dq krb5kdc[3870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:26 dist-test-slave-25dq krb5kdc[3870](info): set up 2 sockets
Sep 27 21:39:26 dist-test-slave-25dq krb5kdc[3870](info): commencing operation
krb5kdc: starting...
W20260927 21:39:27.272908 810 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:27 dist-test-slave-25dq krb5kdc[3870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:27.480711 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:46603
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:46603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:27.808609 3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:27.808897 3883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:27.809073 3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:27.816336 3883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:27.816468 3883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:27.816571 3883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:27.816649 3883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:27.816689 3883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:27.826668 3883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:46603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:46603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:27.828105 3883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:27.829470 3883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:27.841342 3888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:27.841625 3889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:27.841758 3891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:27.842968 3883 server_base.cc:1034] running on GCE node
I20260927 21:39:27.843804 3883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:27.845845 3883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:27.847087 3883 hybrid_clock.cc:648] HybridClock initialized: now 1790545167847028 us; error 51 us; skew 500 ppm
I20260927 21:39:27.847453 3883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:27 dist-test-slave-25dq krb5kdc[3870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:27.850768 3883 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:27.852658 3883 webserver.cc:460] Webserver started at http://127.0.202.190:35001/ using document root <none> and password file <none>
I20260927 21:39:27.853446 3883 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:27.853596 3883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:27.853925 3883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:27.857227 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "e12b57b7f43c475b887d3e9b3a7bcf05"
format_stamp: "Formatted at 2026-09-27 21:39:27 on dist-test-slave-25dq"
server_key: "8d1b770174de73ce156883656bd910a5"
server_key_iv: "69a68a74f1be7750594f527c44009a9e"
server_key_version: "encryptionkey@0"
I20260927 21:39:27.858311 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "e12b57b7f43c475b887d3e9b3a7bcf05"
format_stamp: "Formatted at 2026-09-27 21:39:27 on dist-test-slave-25dq"
server_key: "8d1b770174de73ce156883656bd910a5"
server_key_iv: "69a68a74f1be7750594f527c44009a9e"
server_key_version: "encryptionkey@0"
I20260927 21:39:27.864413 3883 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:39:27.869376 3898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:27.871196 3883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 21:39:27.871443 3883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "e12b57b7f43c475b887d3e9b3a7bcf05"
format_stamp: "Formatted at 2026-09-27 21:39:27 on dist-test-slave-25dq"
server_key: "8d1b770174de73ce156883656bd910a5"
server_key_iv: "69a68a74f1be7750594f527c44009a9e"
server_key_version: "encryptionkey@0"
I20260927 21:39:27.871683 3883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:27.894912 3883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:27.899700 3883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:27.900017 3883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:27.919204 3883 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:46603
I20260927 21:39:27.919245 3949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:46603 every 8 connection(s)
I20260927 21:39:27.920678 3883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:27.924731 3950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:27.930428 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 3883
I20260927 21:39:27.930636 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:27.931440 810 external_mini_cluster.cc:1428] Setting key a7315d2b5ef459e43f42a94f41f33a8f
I20260927 21:39:27.936002 3950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05: Bootstrap starting.
I20260927 21:39:27.939901 3950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:27.941200 3950 log.cc:824] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:27.944952 3950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05: No bootstrap required, opened a new log
Sep 27 21:39:27 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:27.953980 3950 raft_consensus.cc:348] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46603 } }
I20260927 21:39:27.954418 3950 raft_consensus.cc:374] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:27.954545 3950 raft_consensus.cc:729] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12b57b7f43c475b887d3e9b3a7bcf05, State: Initialized, Role: FOLLOWER
I20260927 21:39:27.955226 3950 consensus_queue.cc:260] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46603 } }
I20260927 21:39:27.955487 3950 raft_consensus.cc:388] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:27.955644 3950 raft_consensus.cc:482] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:27.955829 3950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:27.956166 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:27.933648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.0.202.190:46603)
0927 21:39:27.934478 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:27.934491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:27.934532 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:39:27.936249 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:27.936273 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:27.936743 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:27.937068 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:27.938864 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:27.940534 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:27.942262 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:27.943045 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:27.943190 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:27.948253 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:27.948263 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:27.948280 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:27.948389 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:27.951679 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:27.952445 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:27.952453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:27.952460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:27.952646 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:27.953054 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:27.953059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:27.953063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:27.953635 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:27.953947 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:27.954409 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 21:39:27.954850 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":419,"threads_started":1}
I20260927 21:39:27.958289 3950 raft_consensus.cc:504] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46603 } }
I20260927 21:39:27.958810 3950 leader_election.cc:302] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e12b57b7f43c475b887d3e9b3a7bcf05; no voters:
I20260927 21:39:27.959448 3950 leader_election.cc:288] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:27.959599 3955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:27.960539 3955 raft_consensus.cc:686] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [term 1 LEADER]: Becoming Leader. State: Replica: e12b57b7f43c475b887d3e9b3a7bcf05, State: Running, Role: LEADER
I20260927 21:39:27.961107 3955 consensus_queue.cc:237] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46603 } }
I20260927 21:39:27.961861 3950 sys_catalog.cc:564] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:27.963866 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12b57b7f43c475b887d3e9b3a7bcf05. Latest consensus state: current_term: 1 leader_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46603 } } }
I20260927 21:39:27.964270 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:27.969363 3966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:27.969432 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12b57b7f43c475b887d3e9b3a7bcf05" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46603 } } }
I20260927 21:39:27.969785 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:27.974262 3966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:27.983418 3966 catalog_manager.cc:1347] Generated new cluster ID: d867d1c69d49469b9f76f7de43806ff1
I20260927 21:39:27.983556 3966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:28.008407 3966 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:28.010468 3966 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:28.031837 3966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e12b57b7f43c475b887d3e9b3a7bcf05: Generated new TSK 0
I20260927 21:39:28.032999 3966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:28.093564 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46603
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:28.430764 3976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:28.431084 3976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:28.431278 3976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:28.439888 3976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:28.440006 3976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:28.440203 3976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:28.451056 3976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:28.452543 3976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:28.453953 3976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:28.467536 3984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:28.467653 3982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:28.468225 3981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:28.468684 3976 server_base.cc:1034] running on GCE node
I20260927 21:39:28.469282 3976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:28.470206 3976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:28.471437 3976 hybrid_clock.cc:648] HybridClock initialized: now 1790545168471374 us; error 57 us; skew 500 ppm
I20260927 21:39:28.471809 3976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:28 dist-test-slave-25dq krb5kdc[3870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:28.475214 3976 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:28.477174 3976 webserver.cc:460] Webserver started at http://127.0.202.129:37493/ using document root <none> and password file <none>
I20260927 21:39:28.477907 3976 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:28.478056 3976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:28.478369 3976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:28.482054 3976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "f1984c89ebd94aad839040be893dca5f"
format_stamp: "Formatted at 2026-09-27 21:39:28 on dist-test-slave-25dq"
server_key: "867481c92048de77e5f3de3a2f8ffc03"
server_key_iv: "92d26951c5606fc5f04319cba7c77cc0"
server_key_version: "encryptionkey@0"
I20260927 21:39:28.482966 3976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "f1984c89ebd94aad839040be893dca5f"
format_stamp: "Formatted at 2026-09-27 21:39:28 on dist-test-slave-25dq"
server_key: "867481c92048de77e5f3de3a2f8ffc03"
server_key_iv: "92d26951c5606fc5f04319cba7c77cc0"
server_key_version: "encryptionkey@0"
I20260927 21:39:28.488560 3976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:39:28.492170 3991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:28.493786 3976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:28.493996 3976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "f1984c89ebd94aad839040be893dca5f"
format_stamp: "Formatted at 2026-09-27 21:39:28 on dist-test-slave-25dq"
server_key: "867481c92048de77e5f3de3a2f8ffc03"
server_key_iv: "92d26951c5606fc5f04319cba7c77cc0"
server_key_version: "encryptionkey@0"
I20260927 21:39:28.494230 3976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:28.508132 3976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:28.512930 3976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:28.513309 3976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:28.514487 3976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:28.516201 3976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:28.516328 3976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:28.516439 3976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:28.516486 3976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:28.549120 3976 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41945
I20260927 21:39:28.549162 4103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41945 every 8 connection(s)
I20260927 21:39:28.550369 3976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:28.553254 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 3976
I20260927 21:39:28.553484 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:28.554106 810 external_mini_cluster.cc:1428] Setting key ac5eabe30a62f45dcfd9f41005a5d629
Sep 27 21:39:28 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:39:28.574159 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:28.553138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:58857 (local address 127.0.202.190:46603)
0927 21:39:28.553651 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:28.553658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:28.554809 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 21:39:28.556729 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:28.556737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:28.556832 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:28.556956 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:28.558663 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:28.559849 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:28.562240 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:28.562655 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:28.562773 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:28.567640 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:28.567646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:28.567649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:28.567685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:28.570352 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:28.571618 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:28.571629 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:28.571632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:28.571782 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:28.572323 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:28.572329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:28.572332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:28.572605 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:28.572771 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:28.573544 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 21:39:28.573777 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260927 21:39:28.575286 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:28.553780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46603 (local address 127.0.202.129:58857)
0927 21:39:28.554642 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:28.554688 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:28.556458 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:28.557267 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:28.557276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:28.557811 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:28.558473 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:28.558492 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:28.560049 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:28.560053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:28.561934 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:28.561956 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:28.562918 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:28.562929 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:28.563343 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:28.564372 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:28.564407 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:28.567370 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:28.570526 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:28.570536 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:28.570557 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:28.571368 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:28.571915 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:28.571920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:28.571924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:28.572168 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:28.572761 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:28.572770 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:28.573152 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 21:39:28.573433 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 21:39:28.574027 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":378,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:39:28.577363 4104 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46603
I20260927 21:39:28.577750 4104 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:28.578523 4104 heartbeater.cc:511] Master 127.0.202.190:46603 requested a full tablet report, sending...
I20260927 21:39:28.581029 3915 ts_manager.cc:194] Registered new tserver with Master: f1984c89ebd94aad839040be893dca5f (127.0.202.129:41945)
I20260927 21:39:28.582805 3915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:58857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:28.605329 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46603
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:28.937989 4110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:28.938277 4110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:28.938441 4110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:28.945824 4110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:28.945940 4110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:28.946123 4110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:28.955909 4110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:28.957315 4110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:28.958654 4110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:28.972182 4115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:28.972244 4116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:28.972242 4118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:28.972870 4110 server_base.cc:1034] running on GCE node
I20260927 21:39:28.973855 4110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:28.974874 4110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:28.976109 4110 hybrid_clock.cc:648] HybridClock initialized: now 1790545168976054 us; error 46 us; skew 500 ppm
I20260927 21:39:28.976576 4110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:28 dist-test-slave-25dq krb5kdc[3870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:28.979728 4110 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:28.981343 4110 webserver.cc:460] Webserver started at http://127.0.202.130:41615/ using document root <none> and password file <none>
I20260927 21:39:28.981999 4110 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:28.982097 4110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:28.982339 4110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:28.985275 4110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "e32e6e5e24a94a1c9156e037eef8f8aa"
format_stamp: "Formatted at 2026-09-27 21:39:28 on dist-test-slave-25dq"
server_key: "65e91caff761db48e989b8ce323eb787"
server_key_iv: "90ecdfacc2472d3d34f80d78935eea16"
server_key_version: "encryptionkey@0"
I20260927 21:39:28.985968 4110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "e32e6e5e24a94a1c9156e037eef8f8aa"
format_stamp: "Formatted at 2026-09-27 21:39:28 on dist-test-slave-25dq"
server_key: "65e91caff761db48e989b8ce323eb787"
server_key_iv: "90ecdfacc2472d3d34f80d78935eea16"
server_key_version: "encryptionkey@0"
I20260927 21:39:28.991200 4110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:39:28.995054 4125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:28.996687 4110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:39:28.996860 4110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "e32e6e5e24a94a1c9156e037eef8f8aa"
format_stamp: "Formatted at 2026-09-27 21:39:28 on dist-test-slave-25dq"
server_key: "65e91caff761db48e989b8ce323eb787"
server_key_iv: "90ecdfacc2472d3d34f80d78935eea16"
server_key_version: "encryptionkey@0"
I20260927 21:39:28.997045 4110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:29.024712 4110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:29.029248 4110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:29.029578 4110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:29.030568 4110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:29.032068 4110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:29.032166 4110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:29.032369 4110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:29.032421 4110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 21:39:29.069293 4110 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:34245
I20260927 21:39:29.069427 4237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:34245 every 8 connection(s)
I20260927 21:39:29.070665 4110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:29.079824 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 4110
I20260927 21:39:29.080035 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:29.080595 810 external_mini_cluster.cc:1428] Setting key 4fc33685dd4bf162c3a392e418149dad
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:29.092012 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.073410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:38285 (local address 127.0.202.190:46603)
0927 21:39:29.073611 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:29.073616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:29.074744 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 21:39:29.076860 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:29.076867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:29.076971 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:29.077092 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:29.078718 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.079637 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.081666 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.082221 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.082388 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.086202 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:29.086208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:29.086211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:29.086244 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:29.088919 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.089695 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.089699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.089701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.089830 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.090282 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.090289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.090297 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.090559 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:29.090723 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:29.091429 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 21:39:29.091679 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:39:29.093077 4240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.073818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46603 (local address 127.0.202.130:38285)
0927 21:39:29.074572 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:29.074614 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:29.076560 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:29.077351 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:29.077361 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:29.077919 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:29.078530 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:29.078548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.079810 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.079814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:29.081474 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:29.081490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.082503 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.082510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:29.082782 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.083643 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:29.083671 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:29.085938 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:29.089078 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:29.089088 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:29.089107 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:29.089546 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:29.089951 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:29.089957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:29.089961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:29.090160 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:29.090694 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:29.090703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:29.091054 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 21:39:29.091329 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 21:39:29.091876 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":288,"threads_started":1}
I20260927 21:39:29.094576 4238 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46603
I20260927 21:39:29.094934 4238 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:29.095664 4238 heartbeater.cc:511] Master 127.0.202.190:46603 requested a full tablet report, sending...
I20260927 21:39:29.097255 3915 ts_manager.cc:194] Registered new tserver with Master: e32e6e5e24a94a1c9156e037eef8f8aa (127.0.202.130:34245)
I20260927 21:39:29.098649 3915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:38285
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:29.128552 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46603
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:29.462718 4243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:29.463052 4243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:29.463189 4243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:29.470706 4243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:29.470819 4243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:29.470934 4243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:29.480727 4243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:29.482203 4243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:29.483551 4243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:29.498097 4248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:29.498027 4249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:29.498055 4251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:29.498988 4243 server_base.cc:1034] running on GCE node
I20260927 21:39:29.499580 4243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:29.500624 4243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:29.501844 4243 hybrid_clock.cc:648] HybridClock initialized: now 1790545169501781 us; error 66 us; skew 500 ppm
I20260927 21:39:29.502259 4243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:29.505432 4243 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:29.507051 4243 webserver.cc:460] Webserver started at http://127.0.202.131:42589/ using document root <none> and password file <none>
I20260927 21:39:29.507758 4243 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:29.507860 4243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:29.508101 4243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:29.511072 4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "2857fa9f9f424403ab016265ece54581"
format_stamp: "Formatted at 2026-09-27 21:39:29 on dist-test-slave-25dq"
server_key: "0cdcfc169e00d49981340672ce6ec05a"
server_key_iv: "25544c617d721c9e7e4737ab157e3664"
server_key_version: "encryptionkey@0"
I20260927 21:39:29.511806 4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "2857fa9f9f424403ab016265ece54581"
format_stamp: "Formatted at 2026-09-27 21:39:29 on dist-test-slave-25dq"
server_key: "0cdcfc169e00d49981340672ce6ec05a"
server_key_iv: "25544c617d721c9e7e4737ab157e3664"
server_key_version: "encryptionkey@0"
I20260927 21:39:29.517223 4243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260927 21:39:29.521071 4258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:29.522616 4243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:29.522804 4243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "2857fa9f9f424403ab016265ece54581"
format_stamp: "Formatted at 2026-09-27 21:39:29 on dist-test-slave-25dq"
server_key: "0cdcfc169e00d49981340672ce6ec05a"
server_key_iv: "25544c617d721c9e7e4737ab157e3664"
server_key_version: "encryptionkey@0"
I20260927 21:39:29.522989 4243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:29.544806 4243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:29.549978 4243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:29.550320 4243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:29.551586 4243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:29.553298 4243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:29.553390 4243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:29.553498 4243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:29.553553 4243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:29.586308 4104 heartbeater.cc:503] Master 127.0.202.190:46603 was elected leader, sending a full tablet report...
I20260927 21:39:29.588466 4243 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:43737
I20260927 21:39:29.588486 4370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:43737 every 8 connection(s)
I20260927 21:39:29.589818 4243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:29.599421 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 4243
I20260927 21:39:29.599638 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:29.600065 810 external_mini_cluster.cc:1428] Setting key 26f6d63cb42afeb3ab1e2c58e444ea70
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:29.612286 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.592894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:46533 (local address 127.0.202.190:46603)
0927 21:39:29.593393 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:29.593398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:29.594641 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 21:39:29.596740 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:29.596746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:29.596854 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:29.596983 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:29.598762 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.599751 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.602045 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.602825 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.602958 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.606606 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:29.606611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:29.606614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:29.606662 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:29.609350 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.610091 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.610095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.610097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.610209 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.610629 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.610633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.610635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.610915 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:29.611031 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:29.611841 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 21:39:29.611998 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":242,"threads_started":1}
I20260927 21:39:29.613444 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.593573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46603 (local address 127.0.202.131:46533)
0927 21:39:29.594453 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:29.594501 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:29.596445 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:29.597251 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:29.597261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:29.597878 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:29.598552 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:29.598573 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.599928 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.599932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:29.601889 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:29.601903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.603074 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.603082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:29.603418 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.604186 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:29.604213 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:29.606405 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:29.609484 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:29.609491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:29.609505 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:29.609978 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:29.610317 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:29.610321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:29.610323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:29.610500 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:29.611065 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:29.611072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:29.611440 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 21:39:29.611730 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 21:39:29.612194 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":331,"threads_started":1}
I20260927 21:39:29.615005 4371 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46603
I20260927 21:39:29.615350 4371 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:29.616216 4371 heartbeater.cc:511] Master 127.0.202.190:46603 requested a full tablet report, sending...
I20260927 21:39:29.617734 3914 ts_manager.cc:194] Registered new tserver with Master: 2857fa9f9f424403ab016265ece54581 (127.0.202.131:43737)
I20260927 21:39:29.618786 3914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:46533
I20260927 21:39:29.629065 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545169, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545169, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:29.679570 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.663828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.0.202.190:46603)
0927 21:39:29.664141 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:29.664151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:29.664762 (+ 611us) server_negotiation.cc:407] Connection header received
0927 21:39:29.664997 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:29.665014 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:29.665111 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:29.665243 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:29.666894 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.667720 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.669158 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.669607 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.669761 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.673955 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:29.673960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:29.673964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:29.674000 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:29.676928 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.677590 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.677600 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.677602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.677721 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.678114 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.678118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.678121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.678459 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:29.678714 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:29.679017 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:39:29.679265 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 21:39:29.680148 3914 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36338
I20260927 21:39:29.682039 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3976
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545169, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:29.699170 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.683228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.0.202.129:41945)
0927 21:39:29.683852 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:29.683859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:29.683884 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:39:29.683937 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:29.683943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:29.684153 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:29.684277 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:29.685835 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.686870 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.688059 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.688707 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.688805 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.693043 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:29.693058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:29.693073 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:29.693122 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:29.696058 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.696989 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.696996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.696998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.697131 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.697533 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.697536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.697538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.697979 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:29.698215 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:29.698479 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:39:29.698734 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":313,"threads_started":1}
W20260927 21:39:29.700572 4099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51046
W20260927 21:39:29.701975 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:29.702235 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 3976
I20260927 21:39:29.719074 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4110
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545169, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:39:29.735755 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.720021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.0.202.130:34245)
0927 21:39:29.720587 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:29.720595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:29.720618 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:39:29.720684 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:29.720690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:29.720909 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:29.721079 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:29.722342 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.723364 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.724744 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.725305 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.725459 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.729793 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:29.729805 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:29.729820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:29.729855 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:29.732964 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.733646 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.733659 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.733661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.733779 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.734146 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.734153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.734160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.734488 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:29.734820 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:29.735146 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 21:39:29.735419 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":293,"threads_started":1}
W20260927 21:39:29.737104 4233 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39756
W20260927 21:39:29.738175 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4110: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:29.738337 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 4110
I20260927 21:39:29.754936 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4243
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[3870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545169, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:39:29.773168 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:29.756026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.0.202.131:43737)
0927 21:39:29.756787 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:29.756795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:29.756827 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:39:29.756890 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:29.756897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:29.757116 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:29.757258 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:29.758678 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.759837 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.761353 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:29.762001 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:29.762108 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:29.767110 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:29.767122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:29.767138 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:29.767180 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:29.770395 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.771062 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.771070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.771072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.771211 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:29.771543 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:29.771546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:29.771549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:29.771904 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:29.772138 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:29.772498 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 21:39:29.772734 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":422,"threads_started":1}
W20260927 21:39:29.774641 4366 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37748
W20260927 21:39:29.775858 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4243: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:39:29.776011 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 4243
I20260927 21:39:29.791688 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 3883
I20260927 21:39:29.793049 3945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:29.898700 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 3883
2026-09-27T21:39:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3737 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[4389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[4389](info): set up 2 sockets
Sep 27 21:39:29 dist-test-slave-25dq krb5kdc[4389](info): commencing operation
krb5kdc: starting...
W20260927 21:39:31.020815 810 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:31 dist-test-slave-25dq krb5kdc[4389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:31.233956 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:46501
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:31.563930 4402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:31.564258 4402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:31.564431 4402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:31.571794 4402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:31.571928 4402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:31.571981 4402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:31.572038 4402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:31.572082 4402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:31.582221 4402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:31.583536 4402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:31.584829 4402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:31.597668 4410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:31.597640 4408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:31.597767 4407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:31.598131 4402 server_base.cc:1034] running on GCE node
I20260927 21:39:31.599483 4402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:31.601217 4402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:31.602515 4402 hybrid_clock.cc:648] HybridClock initialized: now 1790545171602427 us; error 66 us; skew 500 ppm
I20260927 21:39:31.602926 4402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:31 dist-test-slave-25dq krb5kdc[4389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:31.606487 4402 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:31.608187 4402 webserver.cc:460] Webserver started at http://127.0.202.190:40819/ using document root <none> and password file <none>
I20260927 21:39:31.608959 4402 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:31.609069 4402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:31.609367 4402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:31.612634 4402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "20e4155ab59c4a13921b11eb442e4613"
format_stamp: "Formatted at 2026-09-27 21:39:31 on dist-test-slave-25dq"
server_key: "a0bf0158a717d781d4c3364871c683d6"
server_key_iv: "44a50f5b46d39e0f452aeb193e0b6df5"
server_key_version: "encryptionkey@0"
I20260927 21:39:31.613479 4402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "20e4155ab59c4a13921b11eb442e4613"
format_stamp: "Formatted at 2026-09-27 21:39:31 on dist-test-slave-25dq"
server_key: "a0bf0158a717d781d4c3364871c683d6"
server_key_iv: "44a50f5b46d39e0f452aeb193e0b6df5"
server_key_version: "encryptionkey@0"
I20260927 21:39:31.619493 4402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 21:39:31.623612 4417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:31.625209 4402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:31.625479 4402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "20e4155ab59c4a13921b11eb442e4613"
format_stamp: "Formatted at 2026-09-27 21:39:31 on dist-test-slave-25dq"
server_key: "a0bf0158a717d781d4c3364871c683d6"
server_key_iv: "44a50f5b46d39e0f452aeb193e0b6df5"
server_key_version: "encryptionkey@0"
I20260927 21:39:31.625674 4402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:31.643898 4402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:31.649037 4402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:31.649420 4402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:31.670272 4402 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:46501
I20260927 21:39:31.670320 4468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:46501 every 8 connection(s)
I20260927 21:39:31.671674 4402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:31.674209 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 4402
I20260927 21:39:31.674499 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:31.675271 810 external_mini_cluster.cc:1428] Setting key 8a952b728d3dfdabfee91c625beca9fc
I20260927 21:39:31.675849 4469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:31.691712 4469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613: Bootstrap starting.
Sep 27 21:39:31 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:31.696416 4469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:31.698032 4469 log.cc:824] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:31.702086 4469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613: No bootstrap required, opened a new log
I20260927 21:39:31.704412 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:31.677502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.202.190:46501)
0927 21:39:31.678300 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:31.678317 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:31.678369 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:39:31.680816 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:31.680850 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:31.681447 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:31.681957 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:31.683379 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:31.685484 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:31.687350 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:31.688815 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:31.689077 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:31.693791 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:31.693807 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:31.693831 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:31.693889 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:31.698450 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:31.699704 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:31.699717 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:31.699729 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:31.699961 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:31.700694 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:31.700702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:31.700708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:31.701490 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:31.701788 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:31.702322 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 21:39:31.702846 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":336,"threads_started":1}
I20260927 21:39:31.711578 4469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e4155ab59c4a13921b11eb442e4613" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46501 } }
I20260927 21:39:31.711952 4469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:31.712033 4469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e4155ab59c4a13921b11eb442e4613, State: Initialized, Role: FOLLOWER
I20260927 21:39:31.712690 4469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e4155ab59c4a13921b11eb442e4613" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46501 } }
I20260927 21:39:31.712937 4469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:31.713076 4469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:31.713281 4469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:31.715447 4469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e4155ab59c4a13921b11eb442e4613" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46501 } }
I20260927 21:39:31.715952 4469 leader_election.cc:302] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e4155ab59c4a13921b11eb442e4613; no voters:
I20260927 21:39:31.716661 4469 leader_election.cc:288] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:31.716777 4474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:31.717504 4474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [term 1 LEADER]: Becoming Leader. State: Replica: 20e4155ab59c4a13921b11eb442e4613, State: Running, Role: LEADER
I20260927 21:39:31.718115 4474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e4155ab59c4a13921b11eb442e4613" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46501 } }
I20260927 21:39:31.719038 4469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:31.721174 4475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e4155ab59c4a13921b11eb442e4613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e4155ab59c4a13921b11eb442e4613" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46501 } } }
I20260927 21:39:31.721269 4476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e4155ab59c4a13921b11eb442e4613. Latest consensus state: current_term: 1 leader_uuid: "20e4155ab59c4a13921b11eb442e4613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e4155ab59c4a13921b11eb442e4613" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46501 } } }
I20260927 21:39:31.721562 4475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:31.721630 4476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:31.725525 4479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:31.731112 4479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:31.742779 4479 catalog_manager.cc:1347] Generated new cluster ID: 7eb32449eab04310884303b179a79116
I20260927 21:39:31.742918 4479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:31.775772 4479 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:31.777418 4479 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:31.791918 4479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e4155ab59c4a13921b11eb442e4613: Generated new TSK 0
I20260927 21:39:31.793159 4479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:31.848565 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46501
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:32.191205 4495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:32.191520 4495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:32.191690 4495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:32.199321 4495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:32.199434 4495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:32.199548 4495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:32.210467 4495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:32.211922 4495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:32.213940 4495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:32.228905 4500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:32.228881 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:32.229307 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:32.230425 4495 server_base.cc:1034] running on GCE node
I20260927 21:39:32.231084 4495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:32.232071 4495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:32.233330 4495 hybrid_clock.cc:648] HybridClock initialized: now 1790545172233272 us; error 65 us; skew 500 ppm
I20260927 21:39:32.233662 4495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:32 dist-test-slave-25dq krb5kdc[4389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:32.237335 4495 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:32.238931 4495 webserver.cc:460] Webserver started at http://127.0.202.129:43891/ using document root <none> and password file <none>
I20260927 21:39:32.239750 4495 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:32.239910 4495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:32.240266 4495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:32.243645 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "8b8734d2e6974d91988e5c2edc01c7b9"
format_stamp: "Formatted at 2026-09-27 21:39:32 on dist-test-slave-25dq"
server_key: "fdebe3347935d10826a30d7a759cc529"
server_key_iv: "6374fda5b1090fe7db77bad16b6ff0c7"
server_key_version: "encryptionkey@0"
I20260927 21:39:32.244665 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "8b8734d2e6974d91988e5c2edc01c7b9"
format_stamp: "Formatted at 2026-09-27 21:39:32 on dist-test-slave-25dq"
server_key: "fdebe3347935d10826a30d7a759cc529"
server_key_iv: "6374fda5b1090fe7db77bad16b6ff0c7"
server_key_version: "encryptionkey@0"
I20260927 21:39:32.250543 4495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:39:32.254506 4510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:32.256501 4495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:39:32.256711 4495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "8b8734d2e6974d91988e5c2edc01c7b9"
format_stamp: "Formatted at 2026-09-27 21:39:32 on dist-test-slave-25dq"
server_key: "fdebe3347935d10826a30d7a759cc529"
server_key_iv: "6374fda5b1090fe7db77bad16b6ff0c7"
server_key_version: "encryptionkey@0"
I20260927 21:39:32.256956 4495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:32.273604 4495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:32.279184 4495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:32.279574 4495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:32.280805 4495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:32.282859 4495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:32.282992 4495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:32.283159 4495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:32.283245 4495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:32.321911 4495 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:35387
I20260927 21:39:32.322244 4622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:35387 every 8 connection(s)
I20260927 21:39:32.323400 4495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:32.330133 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 4495
I20260927 21:39:32.330401 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:32.331076 810 external_mini_cluster.cc:1428] Setting key d7c1c91e531ffb220c8927505fb6ef03
Sep 27 21:39:32 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:32.347064 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:32.326321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:45391 (local address 127.0.202.190:46501)
0927 21:39:32.326807 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:32.326826 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:32.327986 (+ 1160us) server_negotiation.cc:407] Connection header received
0927 21:39:32.329947 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:32.329952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:32.330046 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:32.330165 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:32.331951 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.332847 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.334544 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.334943 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.335079 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:32.339155 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:32.339161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:32.339164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:32.339202 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:32.342982 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:32.344164 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:32.344170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:32.344174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:32.344385 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:32.345011 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:32.345019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:32.345024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:32.345400 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:32.345552 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:32.346392 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 21:39:32.346628 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":255,"threads_started":1}
I20260927 21:39:32.348156 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:32.327028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46501 (local address 127.0.202.129:45391)
0927 21:39:32.327792 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:32.327855 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:32.329660 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:32.330457 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:32.330487 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:32.331034 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:32.331761 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:32.331778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.332995 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.333000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:32.334393 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:32.334405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.335172 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.335179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:32.335433 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:32.336138 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:32.336162 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:32.338863 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:32.343225 (+ 4362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:32.343248 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:32.343269 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:32.343967 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:32.344521 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:32.344527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:32.344531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:32.344851 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:32.345541 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:32.345550 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:32.345991 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 21:39:32.346291 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 21:39:32.346898 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":336,"threads_started":1}
I20260927 21:39:32.350069 4623 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46501
I20260927 21:39:32.350438 4623 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:32.351421 4623 heartbeater.cc:511] Master 127.0.202.190:46501 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:39:32.353972 4434 ts_manager.cc:194] Registered new tserver with Master: 8b8734d2e6974d91988e5c2edc01c7b9 (127.0.202.129:35387)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:39:32.355738 4434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:45391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:32.381711 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46501
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:32.720340 4629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:32.720706 4629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:32.720880 4629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:32.728158 4629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:32.728264 4629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:32.728473 4629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:32.738871 4629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:32.740211 4629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:32.741604 4629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:32.755080 4634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:32.755097 4635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:32.755195 4637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:32.756006 4629 server_base.cc:1034] running on GCE node
I20260927 21:39:32.756676 4629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:32.758332 4629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:32.759562 4629 hybrid_clock.cc:648] HybridClock initialized: now 1790545172759511 us; error 60 us; skew 500 ppm
I20260927 21:39:32.759887 4629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:32 dist-test-slave-25dq krb5kdc[4389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:32.763356 4629 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:32.764997 4629 webserver.cc:460] Webserver started at http://127.0.202.130:36725/ using document root <none> and password file <none>
I20260927 21:39:32.765713 4629 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:32.765864 4629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:32.766172 4629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:32.769255 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "7733127b454a4d21947b25a401947541"
format_stamp: "Formatted at 2026-09-27 21:39:32 on dist-test-slave-25dq"
server_key: "71182b25556ffe9749268179c9df7f6f"
server_key_iv: "f861732fff3b7bf3bb2cafb04c12b5a0"
server_key_version: "encryptionkey@0"
I20260927 21:39:32.770063 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "7733127b454a4d21947b25a401947541"
format_stamp: "Formatted at 2026-09-27 21:39:32 on dist-test-slave-25dq"
server_key: "71182b25556ffe9749268179c9df7f6f"
server_key_iv: "f861732fff3b7bf3bb2cafb04c12b5a0"
server_key_version: "encryptionkey@0"
I20260927 21:39:32.775463 4629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 21:39:32.779405 4644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:32.780890 4629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:32.781076 4629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "7733127b454a4d21947b25a401947541"
format_stamp: "Formatted at 2026-09-27 21:39:32 on dist-test-slave-25dq"
server_key: "71182b25556ffe9749268179c9df7f6f"
server_key_iv: "f861732fff3b7bf3bb2cafb04c12b5a0"
server_key_version: "encryptionkey@0"
I20260927 21:39:32.781289 4629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:32.803282 4629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:32.808079 4629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:32.808483 4629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:32.809574 4629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:32.811048 4629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:32.811121 4629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:32.811208 4629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:32.811252 4629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:32.846339 4629 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:45215
I20260927 21:39:32.846374 4756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:45215 every 8 connection(s)
I20260927 21:39:32.847698 4629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:32.853245 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 4629
I20260927 21:39:32.853487 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:32.854115 810 external_mini_cluster.cc:1428] Setting key 5b32010f7f45d4bd630cab53e3f55545
Sep 27 21:39:32 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:32.874058 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:32.850678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:53767 (local address 127.0.202.190:46501)
0927 21:39:32.851130 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:32.851134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:32.852063 (+ 929us) server_negotiation.cc:407] Connection header received
0927 21:39:32.855008 (+ 2945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:32.855015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:32.855131 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:32.855281 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:32.857794 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.858589 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.861747 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.862231 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.862383 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:32.866249 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:32.866259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:32.866267 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:32.866314 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:32.870049 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:32.870900 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:32.870908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:32.870915 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:32.871110 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:32.871565 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:32.871572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:32.871578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:32.871976 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:32.872149 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:32.873240 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0927 21:39:32.873533 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":254,"threads_started":1}
I20260927 21:39:32.874979 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:32.851014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46501 (local address 127.0.202.130:53767)
0927 21:39:32.851868 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:32.851915 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:32.854624 (+ 2709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:32.855573 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:32.855587 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:32.856463 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:32.857566 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:32.857596 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.858815 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.858824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:32.861548 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:32.861567 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:32.862508 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:32.862517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:32.862885 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:32.863711 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:32.863739 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:32.865960 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:32.870229 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:32.870237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:32.870252 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:32.870750 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:32.871242 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:32.871245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:32.871248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:32.871434 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:32.872184 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:32.872193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:32.872792 (+ 599us) client_negotiation.cc:769] Sending connection context
0927 21:39:32.873080 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 21:39:32.873609 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:39:32.876860 4757 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46501
I20260927 21:39:32.877246 4757 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:32.878096 4757 heartbeater.cc:511] Master 127.0.202.190:46501 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:39:32.880072 4434 ts_manager.cc:194] Registered new tserver with Master: 7733127b454a4d21947b25a401947541 (127.0.202.130:45215)
I20260927 21:39:32.881021 4434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:53767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:32.908773 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46501
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:33.250998 4763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:33.251291 4763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:33.251387 4763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:33.258731 4763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:33.258844 4763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:33.258960 4763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:33.269343 4763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:33.270594 4763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:33.271931 4763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:33.285739 4771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:33.285638 4768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:33.285878 4769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:33.286653 4763 server_base.cc:1034] running on GCE node
I20260927 21:39:33.287274 4763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:33.288223 4763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:33.289479 4763 hybrid_clock.cc:648] HybridClock initialized: now 1790545173289408 us; error 63 us; skew 500 ppm
I20260927 21:39:33.289817 4763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:33 dist-test-slave-25dq krb5kdc[4389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:33.298055 4763 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:33.299810 4763 webserver.cc:460] Webserver started at http://127.0.202.131:39779/ using document root <none> and password file <none>
I20260927 21:39:33.300674 4763 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:33.300837 4763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:33.301180 4763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:33.304456 4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "51da3a7dec9f4ffeaafa43f8e68a4718"
format_stamp: "Formatted at 2026-09-27 21:39:33 on dist-test-slave-25dq"
server_key: "9fa60ef5cfcefb81a715af70a3f6d7ff"
server_key_iv: "2ff8d23a010781ed6af1d5c3d8c3f72d"
server_key_version: "encryptionkey@0"
I20260927 21:39:33.305335 4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "51da3a7dec9f4ffeaafa43f8e68a4718"
format_stamp: "Formatted at 2026-09-27 21:39:33 on dist-test-slave-25dq"
server_key: "9fa60ef5cfcefb81a715af70a3f6d7ff"
server_key_iv: "2ff8d23a010781ed6af1d5c3d8c3f72d"
server_key_version: "encryptionkey@0"
I20260927 21:39:33.311410 4763 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:39:33.315181 4778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:33.316852 4763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:33.317034 4763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "51da3a7dec9f4ffeaafa43f8e68a4718"
format_stamp: "Formatted at 2026-09-27 21:39:33 on dist-test-slave-25dq"
server_key: "9fa60ef5cfcefb81a715af70a3f6d7ff"
server_key_iv: "2ff8d23a010781ed6af1d5c3d8c3f72d"
server_key_version: "encryptionkey@0"
I20260927 21:39:33.317247 4763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:33.340564 4763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:33.345175 4763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:33.345480 4763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:33.346807 4763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:33.348627 4763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:33.348731 4763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:33.348853 4763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:33.348909 4763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:33.358860 4623 heartbeater.cc:503] Master 127.0.202.190:46501 was elected leader, sending a full tablet report...
I20260927 21:39:33.383773 4763 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:44319
I20260927 21:39:33.383888 4890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:44319 every 8 connection(s)
I20260927 21:39:33.385171 4763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:33.390933 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 4763
I20260927 21:39:33.391147 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:33.391683 810 external_mini_cluster.cc:1428] Setting key b58c24dfe5e4d1ab8d3f855a89dcfdd5
Sep 27 21:39:33 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:33.407361 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:33.387789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:46981 (local address 127.0.202.190:46501)
0927 21:39:33.388229 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:33.388234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:33.389416 (+ 1182us) server_negotiation.cc:407] Connection header received
0927 21:39:33.391632 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:33.391637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:33.391716 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:33.391879 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:33.393954 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.395014 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.396756 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.397133 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.397230 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:33.400732 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:33.400736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:33.400740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:33.400771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:33.403707 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.404672 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.404709 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.404712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.404828 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.405396 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.405402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.405405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.405694 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:33.405820 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:33.406759 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 21:39:33.406945 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260927 21:39:33.408480 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:33.388467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46501 (local address 127.0.202.131:46981)
0927 21:39:33.389259 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:33.389304 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:33.391320 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:33.392179 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:33.392189 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:33.392960 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:33.393720 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:33.393760 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.395160 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.395169 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:33.396630 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:33.396642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.397367 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.397372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:33.397565 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:33.398174 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:33.398199 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:33.400523 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:33.403863 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:33.403871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:33.403885 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:33.404529 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:33.404986 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:33.404991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:33.404994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:33.405253 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:33.405824 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:33.405834 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:33.406350 (+ 516us) client_negotiation.cc:769] Sending connection context
0927 21:39:33.406605 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 21:39:33.407200 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":289,"threads_started":1}
I20260927 21:39:33.409929 4891 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46501
I20260927 21:39:33.410277 4891 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:33.410944 4891 heartbeater.cc:511] Master 127.0.202.190:46501 requested a full tablet report, sending...
I20260927 21:39:33.412386 4434 ts_manager.cc:194] Registered new tserver with Master: 51da3a7dec9f4ffeaafa43f8e68a4718 (127.0.202.131:44319)
I20260927 21:39:33.413214 4434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:46981
I20260927 21:39:33.419643 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:39:33 dist-test-slave-25dq krb5kdc[4389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:39:33 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:33.470074 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:33.454127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.0.202.129:35387)
0927 21:39:33.454711 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:33.454719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:33.454745 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:39:33.454810 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:33.454817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:33.455028 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:33.455159 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:33.456715 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.457862 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.459102 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.459746 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.459866 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:33.464563 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:33.464573 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:33.464588 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:33.464627 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:33.467385 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.468046 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.468052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.468055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.468168 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.468562 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.468571 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.468573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.468897 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:33.469114 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:33.469491 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 21:39:33.469675 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":308,"threads_started":1}
Sep 27 21:39:33 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:33.490378 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:33.475145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.0.202.190:46501)
0927 21:39:33.475499 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:33.475508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:33.475785 (+ 277us) server_negotiation.cc:407] Connection header received
0927 21:39:33.475991 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:33.476002 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:33.476084 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:33.476205 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:33.477532 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.478325 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.479684 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.480107 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.480272 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:33.484790 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:33.484795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:33.484799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:33.484838 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:33.487590 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.488361 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.488370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.488392 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.488526 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.488915 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.488918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.488920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.489334 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:33.489514 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:33.489785 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 21:39:33.489963 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
W20260927 21:39:33.490947 4434 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39070
I20260927 21:39:33.491942 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4495
I20260927 21:39:33.505463 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:33.493057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.0.202.129:35387)
0927 21:39:33.493284 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:33.493290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:33.493487 (+ 197us) server_negotiation.cc:407] Connection header received
0927 21:39:33.493730 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:33.493734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:33.493797 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:33.493932 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:33.495195 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.495871 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.497161 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.497592 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.497741 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:33.500093 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:33.500098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:33.500102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:33.500147 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:33.502864 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.503537 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.503543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.503547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.503671 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.504058 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.504063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.504066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.504379 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:33.504614 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:33.504918 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:39:33.505140 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 21:39:33.506103 4618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:33.588475 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 4495
I20260927 21:39:33.604679 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4629
Sep 27 21:39:33 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:39:33.621733 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:33.605531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.0.202.130:45215)
0927 21:39:33.606139 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:33.606146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:33.606171 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:39:33.606237 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:33.606244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:33.606507 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:33.606641 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:33.607856 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.609017 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.610460 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.610998 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.611110 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:33.615443 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:33.615455 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:33.615469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:33.615507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:33.618331 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.619005 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.619012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.619014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.619131 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.619735 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.619739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.619741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.620094 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:33.620406 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:33.621049 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 21:39:33.621418 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":359,"threads_started":1}
I20260927 21:39:33.623201 4752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:33.708230 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 4629
I20260927 21:39:33.725070 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4763
Sep 27 21:39:33 dist-test-slave-25dq krb5kdc[4389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:39:33.743153 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:33.726062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.0.202.131:44319)
0927 21:39:33.726644 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:33.726651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:33.726678 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:39:33.726735 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:33.726741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:33.726985 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:33.727112 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:33.729098 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.730298 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.731815 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:33.732525 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:33.732709 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:33.737334 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:33.737347 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:33.737363 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:33.737404 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:33.740506 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.741287 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.741295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.741297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.741431 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:33.741753 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:33.741757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:33.741758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:33.742096 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:33.742377 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:33.742560 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 21:39:33.742757 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":308,"threads_started":1}
I20260927 21:39:33.744673 4886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:33.827960 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 4763
I20260927 21:39:33.843806 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4402
I20260927 21:39:33.845350 4464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:33.953603 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 4402
2026-09-27T21:39:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4055 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:34 dist-test-slave-25dq krb5kdc[4911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:34 dist-test-slave-25dq krb5kdc[4911](info): set up 2 sockets
Sep 27 21:39:34 dist-test-slave-25dq krb5kdc[4911](info): commencing operation
krb5kdc: starting...
W20260927 21:39:35.085232 810 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:35 dist-test-slave-25dq krb5kdc[4911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:35.297271 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44017
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:44017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:35.639012 4924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:35.639371 4924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:35.639516 4924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:35.646940 4924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:35.647090 4924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:35.647195 4924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:35.647308 4924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:35.647387 4924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:35.657724 4924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:44017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:35.659070 4924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:35.660434 4924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:35.672633 4932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:35.672570 4929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:35.672595 4930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:35.673638 4924 server_base.cc:1034] running on GCE node
I20260927 21:39:35.674510 4924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:35.675990 4924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:35.677227 4924 hybrid_clock.cc:648] HybridClock initialized: now 1790545175677138 us; error 73 us; skew 500 ppm
I20260927 21:39:35.677672 4924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:35 dist-test-slave-25dq krb5kdc[4911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:35.681452 4924 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:35.682700 4924 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:35.682823 4924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:35.683328 4924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:35.686815 4924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "19bc45cea4aa404887f36aece65919f2"
format_stamp: "Formatted at 2026-09-27 21:39:35 on dist-test-slave-25dq"
server_key: "cb80a6f7ac2459e09fb402e3a06eebdc"
server_key_iv: "69ab0512480c8bd7bc26f8adfdf8bfa1"
server_key_version: "encryptionkey@0"
I20260927 21:39:35.687783 4924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "19bc45cea4aa404887f36aece65919f2"
format_stamp: "Formatted at 2026-09-27 21:39:35 on dist-test-slave-25dq"
server_key: "cb80a6f7ac2459e09fb402e3a06eebdc"
server_key_iv: "69ab0512480c8bd7bc26f8adfdf8bfa1"
server_key_version: "encryptionkey@0"
I20260927 21:39:35.693513 4924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 21:39:35.698027 4938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:35.699661 4924 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:39:35.699966 4924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "19bc45cea4aa404887f36aece65919f2"
format_stamp: "Formatted at 2026-09-27 21:39:35 on dist-test-slave-25dq"
server_key: "cb80a6f7ac2459e09fb402e3a06eebdc"
server_key_iv: "69ab0512480c8bd7bc26f8adfdf8bfa1"
server_key_version: "encryptionkey@0"
I20260927 21:39:35.700207 4924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:35.743481 4924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:35.748775 4924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:35.749096 4924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:35.769747 4924 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:44017
I20260927 21:39:35.769825 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:44017 every 8 connection(s)
I20260927 21:39:35.770761 4924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:35.774680 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:35.778110 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 4924
I20260927 21:39:35.778357 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:35.779130 810 external_mini_cluster.cc:1428] Setting key e1aa8cdd860e73cab59e28c98a44c1f6
I20260927 21:39:35.787468 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2: Bootstrap starting.
I20260927 21:39:35.790616 4990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:35.791651 4990 log.cc:824] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:39:35 dist-test-slave-25dq krb5kdc[4911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:35.794736 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2: No bootstrap required, opened a new log
I20260927 21:39:35.802709 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:35.781398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.0.202.190:44017)
0927 21:39:35.782188 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:35.782200 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:35.782241 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:39:35.783819 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:35.783842 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:35.784359 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:35.784767 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:35.786331 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:35.787924 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:35.789226 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:35.789883 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:35.790069 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:35.794149 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:35.794160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:35.794178 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:35.794225 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:35.798107 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:35.798768 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:35.798777 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:35.798785 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:35.798992 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:35.799388 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:35.799393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:35.799395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:35.800090 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:35.800418 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:35.800862 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 21:39:35.801327 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":377,"threads_started":1}
I20260927 21:39:35.805486 4990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bc45cea4aa404887f36aece65919f2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44017 } }
I20260927 21:39:35.805927 4990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:35.806030 4990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bc45cea4aa404887f36aece65919f2, State: Initialized, Role: FOLLOWER
I20260927 21:39:35.806612 4990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bc45cea4aa404887f36aece65919f2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44017 } }
I20260927 21:39:35.806833 4990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:35.806984 4990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:35.807152 4990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:35.809321 4990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bc45cea4aa404887f36aece65919f2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44017 } }
I20260927 21:39:35.809995 4990 leader_election.cc:302] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19bc45cea4aa404887f36aece65919f2; no voters:
I20260927 21:39:35.810565 4990 leader_election.cc:288] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:35.810707 4995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:35.811918 4995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [term 1 LEADER]: Becoming Leader. State: Replica: 19bc45cea4aa404887f36aece65919f2, State: Running, Role: LEADER
I20260927 21:39:35.812457 4995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bc45cea4aa404887f36aece65919f2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44017 } }
I20260927 21:39:35.812851 4990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:35.815934 4997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19bc45cea4aa404887f36aece65919f2. Latest consensus state: current_term: 1 leader_uuid: "19bc45cea4aa404887f36aece65919f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bc45cea4aa404887f36aece65919f2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44017 } } }
I20260927 21:39:35.816057 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19bc45cea4aa404887f36aece65919f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bc45cea4aa404887f36aece65919f2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44017 } } }
I20260927 21:39:35.816805 4997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:35.816840 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:35.818009 5002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:35.824951 5002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:35.834272 5002 catalog_manager.cc:1347] Generated new cluster ID: 213f12371d8c4c248739cd0366978d8b
I20260927 21:39:35.834388 5002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:35.851191 5002 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:35.853343 5002 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:35.862253 5002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2: Generated new TSK 0
I20260927 21:39:35.863101 5002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:35.930215 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44017
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:36.283226 5016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:36.283567 5016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:36.283740 5016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:36.290856 5016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:36.290961 5016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:36.291143 5016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:36.301034 5016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:36.302425 5016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:36.303812 5016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:36.316847 5024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:36.316910 5021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:36.316809 5022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:36.317610 5016 server_base.cc:1034] running on GCE node
I20260927 21:39:36.318209 5016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:36.319267 5016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:36.320497 5016 hybrid_clock.cc:648] HybridClock initialized: now 1790545176320443 us; error 53 us; skew 500 ppm
I20260927 21:39:36.320885 5016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:36 dist-test-slave-25dq krb5kdc[4911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:36.324419 5016 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:36.325718 5016 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:36.325836 5016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:36.326182 5016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:36.329749 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "e5158b22d100471aadcff34ae7bcc193"
format_stamp: "Formatted at 2026-09-27 21:39:36 on dist-test-slave-25dq"
server_key: "05634cc26d59bd3e9712856f2a1fea01"
server_key_iv: "243dd6e5822189c43634e1a73f27b6b2"
server_key_version: "encryptionkey@0"
I20260927 21:39:36.330690 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "e5158b22d100471aadcff34ae7bcc193"
format_stamp: "Formatted at 2026-09-27 21:39:36 on dist-test-slave-25dq"
server_key: "05634cc26d59bd3e9712856f2a1fea01"
server_key_iv: "243dd6e5822189c43634e1a73f27b6b2"
server_key_version: "encryptionkey@0"
I20260927 21:39:36.336788 5016 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 21:39:36.340425 5030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:36.341764 5016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:36.341926 5016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "e5158b22d100471aadcff34ae7bcc193"
format_stamp: "Formatted at 2026-09-27 21:39:36 on dist-test-slave-25dq"
server_key: "05634cc26d59bd3e9712856f2a1fea01"
server_key_iv: "243dd6e5822189c43634e1a73f27b6b2"
server_key_version: "encryptionkey@0"
I20260927 21:39:36.342142 5016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:36.355546 5016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:36.360569 5016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:36.360908 5016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:36.368222 5016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:36.370298 5016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:36.370460 5016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:39:36.370589 5016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:36.370666 5016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:36.404932 5016 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:42297
I20260927 21:39:36.404978 5142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:42297 every 8 connection(s)
I20260927 21:39:36.405820 5016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:36.413071 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5016
I20260927 21:39:36.413362 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:36.414062 810 external_mini_cluster.cc:1428] Setting key 2f4966e847739714bd38af450035c02b
Sep 27 21:39:36 dist-test-slave-25dq krb5kdc[4911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:36.426983 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:36.408362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:35525 (local address 127.0.202.190:44017)
0927 21:39:36.408820 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:36.408824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:36.409568 (+ 744us) server_negotiation.cc:407] Connection header received
0927 21:39:36.411331 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:36.411336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:36.411425 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:36.411535 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:36.413162 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.414063 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.415774 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.416337 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.416467 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:36.420085 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:36.420093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:36.420097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:36.420147 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:36.423423 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:36.424408 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:36.424413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:36.424416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:36.424543 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:36.425058 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:36.425063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:36.425065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:36.425429 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:36.425547 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:36.426499 (+ 952us) server_negotiation.cc:299] Negotiation successful
0927 21:39:36.426679 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260927 21:39:36.428481 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:36.408755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44017 (local address 127.0.202.129:35525)
0927 21:39:36.409410 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:36.409453 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:36.411072 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:36.411769 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:36.411777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:36.412278 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:36.413003 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:36.413022 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.414207 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.414212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:36.415598 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:36.415609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.416537 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.416542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:36.416731 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:36.417407 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:36.417433 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:36.419829 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:36.423560 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:36.423572 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:36.423593 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:36.424251 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:36.424672 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:36.424679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:36.424685 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:36.424917 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:36.425556 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:36.425567 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:36.426047 (+ 480us) client_negotiation.cc:769] Sending connection context
0927 21:39:36.426355 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 21:39:36.426956 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":257,"threads_started":1}
I20260927 21:39:36.430213 5143 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44017
I20260927 21:39:36.430557 5143 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:36.431234 5143 heartbeater.cc:511] Master 127.0.202.190:44017 requested a full tablet report, sending...
I20260927 21:39:36.433341 4955 ts_manager.cc:194] Registered new tserver with Master: e5158b22d100471aadcff34ae7bcc193 (127.0.202.129:42297)
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:39:36.434844 4955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:35525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:36.463994 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44017
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:36.801396 5149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:36.801707 5149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:36.801879 5149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:36.810240 5149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:36.810376 5149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:36.810557 5149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:36.821959 5149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:36.823443 5149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:36.824926 5149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:36.839268 5155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:36.839581 5154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:36.839223 5157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:36.840505 5149 server_base.cc:1034] running on GCE node
I20260927 21:39:36.841137 5149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:36.842156 5149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:36.843475 5149 hybrid_clock.cc:648] HybridClock initialized: now 1790545176843356 us; error 56 us; skew 500 ppm
I20260927 21:39:36.843918 5149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:36 dist-test-slave-25dq krb5kdc[4911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:36.848224 5149 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:36.849673 5149 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:36.849781 5149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:36.850047 5149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:36.853235 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "0c9907c8c5c248e7b949cf03e37127c5"
format_stamp: "Formatted at 2026-09-27 21:39:36 on dist-test-slave-25dq"
server_key: "66948f50776649d0991d8b5f092ac85e"
server_key_iv: "c256c18994391fd49fc4aa215c996822"
server_key_version: "encryptionkey@0"
I20260927 21:39:36.854099 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "0c9907c8c5c248e7b949cf03e37127c5"
format_stamp: "Formatted at 2026-09-27 21:39:36 on dist-test-slave-25dq"
server_key: "66948f50776649d0991d8b5f092ac85e"
server_key_iv: "c256c18994391fd49fc4aa215c996822"
server_key_version: "encryptionkey@0"
I20260927 21:39:36.859690 5149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:39:36.863559 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:36.865128 5149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:36.865312 5149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "0c9907c8c5c248e7b949cf03e37127c5"
format_stamp: "Formatted at 2026-09-27 21:39:36 on dist-test-slave-25dq"
server_key: "66948f50776649d0991d8b5f092ac85e"
server_key_iv: "c256c18994391fd49fc4aa215c996822"
server_key_version: "encryptionkey@0"
I20260927 21:39:36.865537 5149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:36.901007 5149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:36.905834 5149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:36.906149 5149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:36.907169 5149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:36.908766 5149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:36.908859 5149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:36.908934 5149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:36.908988 5149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:36.942886 5149 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:41447
I20260927 21:39:36.943010 5275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:41447 every 8 connection(s)
I20260927 21:39:36.943756 5149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:36.944367 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5149
I20260927 21:39:36.944600 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:36.945163 810 external_mini_cluster.cc:1428] Setting key 4cbea57a5d4c63fab337a1752300e274
Sep 27 21:39:36 dist-test-slave-25dq krb5kdc[4911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:39:36.971105 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:36.947182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:55591 (local address 127.0.202.190:44017)
0927 21:39:36.947651 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:36.947656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:36.948808 (+ 1152us) server_negotiation.cc:407] Connection header received
0927 21:39:36.952056 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:36.952061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:36.952130 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:36.952227 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:36.954633 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.955883 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.958021 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.958814 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.958956 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:36.963448 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:36.963457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:36.963466 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:36.963518 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:36.967290 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:36.968195 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:36.968199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:36.968201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:36.968418 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:36.969037 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:36.969044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:36.969049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:36.969488 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:36.969627 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:36.970670 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0927 21:39:36.970838 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":262,"threads_started":1}
I20260927 21:39:36.972983 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:36.947775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44017 (local address 127.0.202.130:55591)
0927 21:39:36.948621 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:36.948679 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:36.951697 (+ 3018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:36.952562 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:36.952585 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:36.953478 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:36.954404 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:36.954436 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.956076 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.956080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:36.957868 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:36.957881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:36.959081 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:36.959087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:36.959417 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:36.960220 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:36.960372 (+ 152us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:36.963133 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:36.967472 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:36.967480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:36.967496 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:36.968039 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:36.968563 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:36.968571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:36.968576 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:36.968798 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:36.969650 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:36.969659 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:36.970186 (+ 527us) client_negotiation.cc:769] Sending connection context
0927 21:39:36.970570 (+ 384us) client_negotiation.cc:240] Negotiation successful
0927 21:39:36.971233 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":299,"threads_started":1}
I20260927 21:39:36.974620 5276 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44017
I20260927 21:39:36.974961 5276 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:36.975638 5276 heartbeater.cc:511] Master 127.0.202.190:44017 requested a full tablet report, sending...
I20260927 21:39:36.977066 4955 ts_manager.cc:194] Registered new tserver with Master: 0c9907c8c5c248e7b949cf03e37127c5 (127.0.202.130:41447)
I20260927 21:39:36.977871 4955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:55591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:37.000397 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44017
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:37.335373 5282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:37.335717 5282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:37.335876 5282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:37.343248 5282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:37.343405 5282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:37.343583 5282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:37.354000 5282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:37.355363 5282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:37.356758 5282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:37.373533 5287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:37.373605 5290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:37.374177 5288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:37.375192 5282 server_base.cc:1034] running on GCE node
I20260927 21:39:37.375875 5282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:37.376983 5282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:37.378223 5282 hybrid_clock.cc:648] HybridClock initialized: now 1790545177378180 us; error 41 us; skew 500 ppm
I20260927 21:39:37.378657 5282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:37 dist-test-slave-25dq krb5kdc[4911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:37.382162 5282 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:37.383458 5282 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:37.383594 5282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:37.383926 5282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:37.386929 5282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "b9a0d89415dd41d0928b24e1eb0acf1c"
format_stamp: "Formatted at 2026-09-27 21:39:37 on dist-test-slave-25dq"
server_key: "0cdd3bc2ccc5c2fd5675fab31a4cf034"
server_key_iv: "5abfc04fc94fc25a7b084cb91c272808"
server_key_version: "encryptionkey@0"
I20260927 21:39:37.387748 5282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "b9a0d89415dd41d0928b24e1eb0acf1c"
format_stamp: "Formatted at 2026-09-27 21:39:37 on dist-test-slave-25dq"
server_key: "0cdd3bc2ccc5c2fd5675fab31a4cf034"
server_key_iv: "5abfc04fc94fc25a7b084cb91c272808"
server_key_version: "encryptionkey@0"
I20260927 21:39:37.393540 5282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 21:39:37.397377 5296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:37.398904 5282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 21:39:37.399083 5282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "b9a0d89415dd41d0928b24e1eb0acf1c"
format_stamp: "Formatted at 2026-09-27 21:39:37 on dist-test-slave-25dq"
server_key: "0cdd3bc2ccc5c2fd5675fab31a4cf034"
server_key_iv: "5abfc04fc94fc25a7b084cb91c272808"
server_key_version: "encryptionkey@0"
I20260927 21:39:37.399305 5282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:37.416917 5282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:37.421532 5282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:37.421808 5282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:37.422816 5282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:37.424173 5282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:37.424299 5282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:37.424551 5282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:37.424639 5282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:37.438429 5143 heartbeater.cc:503] Master 127.0.202.190:44017 was elected leader, sending a full tablet report...
I20260927 21:39:37.458825 5282 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:36845
I20260927 21:39:37.458918 5408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:36845 every 8 connection(s)
I20260927 21:39:37.459673 5282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:37.460426 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5282
I20260927 21:39:37.460644 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:37.461225 810 external_mini_cluster.cc:1428] Setting key 26f711e8e6efe8d77c5fd0993066da1e
Sep 27 21:39:37 dist-test-slave-25dq krb5kdc[4911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:37.480681 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.462807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:58781 (local address 127.0.202.190:44017)
0927 21:39:37.463015 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.463022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.464372 (+ 1350us) server_negotiation.cc:407] Connection header received
0927 21:39:37.466506 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.466511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.466616 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.466720 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:37.468264 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.469090 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.470795 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.471191 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.471343 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.474788 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:37.474792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:37.474796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:37.474826 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:37.477685 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.478404 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.478411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.478415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.478530 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.478929 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.478936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.478939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.479201 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:37.479364 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.480086 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.480346 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:39:37.481658 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.463363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44017 (local address 127.0.202.131:58781)
0927 21:39:37.464197 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:37.464248 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:37.466243 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:37.466980 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:37.466988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:37.467502 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:37.468072 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.468088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.469235 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.469241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.470639 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.470650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.471422 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.471429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.471649 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.472323 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:37.472358 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:37.474604 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:37.477823 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:37.477833 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:37.477852 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:37.478290 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:37.478633 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:37.478638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:37.478642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:37.478822 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:37.479343 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:37.479352 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:37.479720 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 21:39:37.479992 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 21:39:37.480581 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":504,"threads_started":1}
I20260927 21:39:37.483153 5409 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44017
I20260927 21:39:37.483589 5409 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:37.484259 5409 heartbeater.cc:511] Master 127.0.202.190:44017 requested a full tablet report, sending...
I20260927 21:39:37.485775 4955 ts_manager.cc:194] Registered new tserver with Master: b9a0d89415dd41d0928b24e1eb0acf1c (127.0.202.131:36845)
I20260927 21:39:37.486583 4955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:58781
I20260927 21:39:37.490268 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:39:37.508118 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.494816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.0.202.190:44017)
0927 21:39:37.494988 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.494995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.495538 (+ 543us) server_negotiation.cc:407] Connection header received
0927 21:39:37.495646 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.495651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.495713 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.495800 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:37.497270 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.498164 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.499430 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.499834 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.499974 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.502382 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:37.502388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:37.502391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:37.502446 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:37.505664 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.506419 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.506425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.506428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.506560 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.506899 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.506902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.506904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.507209 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:37.507388 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.507669 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.507824 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:39:37.512629 4955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:37.519687 4955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:37.543440 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.532968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:41447 (local address 127.0.0.1:56140)
0927 21:39:37.535774 (+ 2806us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:37.535808 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:37.535944 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:37.536996 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:37.537016 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:37.537051 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:37.537527 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.537540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.539254 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.539260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.541200 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.541210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.542460 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.542466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.542656 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.542707 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:39:37.542846 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:39:37.543039 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2669,"mutex_wait_us":1,"thread_start_us":2438,"threads_started":1}
I20260927 21:39:37.543758 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.533220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.0.202.130:41447)
0927 21:39:37.535341 (+ 2121us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.535354 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.536080 (+ 726us) server_negotiation.cc:407] Connection header received
0927 21:39:37.536179 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.536187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.536502 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.536714 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:37.537703 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.539137 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.541345 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.542258 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.542421 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.542634 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.542883 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.543059 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1930,"thread_start_us":1805,"threads_started":1}
I20260927 21:39:37.545910 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.532463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:36845 (local address 127.0.0.1:58144)
0927 21:39:37.533262 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:37.533404 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:37.533665 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:37.535370 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:37.535381 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:37.535425 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:37.536037 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.536052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.538094 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.538100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.542501 (+ 4401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.542514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.544976 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.544982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.545245 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.545290 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:39:37.545382 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:39:37.545512 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1689,"thread_start_us":669,"threads_started":1}
I20260927 21:39:37.546109 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.532651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.0.202.131:36845)
0927 21:39:37.534511 (+ 1860us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.534526 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.534571 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:39:37.534710 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.534722 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.535070 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.535547 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:37.536256 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.537951 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.542700 (+ 4749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.544751 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.544901 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.545150 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.545519 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.545710 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599,"thread_start_us":1995,"threads_started":1}
I20260927 21:39:37.546530 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.536284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.0.202.129:42297)
0927 21:39:37.536905 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.536913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.537551 (+ 638us) server_negotiation.cc:407] Connection header received
0927 21:39:37.537843 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.537849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.538008 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.538224 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:37.538869 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.540568 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.542333 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.544997 (+ 2664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.545140 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.545405 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.545783 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.546169 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":393,"threads_started":1}
I20260927 21:39:37.546985 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.536581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:42297 (local address 127.0.0.1:50620)
0927 21:39:37.537429 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:37.537449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:37.537655 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:37.538222 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:37.538227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:37.538256 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:37.538671 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.538683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.540729 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.540738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.542176 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.542187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.545335 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.545342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.545535 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.545567 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:39:37.546432 (+ 865us) client_negotiation.cc:240] Negotiation successful
0927 21:39:37.546589 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":503,"threads_started":1}
I20260927 21:39:37.548100 5211 tablet_service.cc:1467] Processing CreateTablet for tablet f810d3150801427ca8b8111e123c9fc3 (DEFAULT_TABLE table=test-table [id=5bfe1d19063a424e8ed2c76024568e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:37.549659 5211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f810d3150801427ca8b8111e123c9fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:37.550328 5344 tablet_service.cc:1467] Processing CreateTablet for tablet f810d3150801427ca8b8111e123c9fc3 (DEFAULT_TABLE table=test-table [id=5bfe1d19063a424e8ed2c76024568e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:37.550575 5078 tablet_service.cc:1467] Processing CreateTablet for tablet f810d3150801427ca8b8111e123c9fc3 (DEFAULT_TABLE table=test-table [id=5bfe1d19063a424e8ed2c76024568e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:37.551502 5344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f810d3150801427ca8b8111e123c9fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:37.551831 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f810d3150801427ca8b8111e123c9fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:37.565163 5427 tablet_bootstrap.cc:492] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Bootstrap starting.
I20260927 21:39:37.566125 5428 tablet_bootstrap.cc:492] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Bootstrap starting.
I20260927 21:39:37.569239 5429 tablet_bootstrap.cc:492] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Bootstrap starting.
I20260927 21:39:37.570763 5428 tablet_bootstrap.cc:654] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:37.570763 5427 tablet_bootstrap.cc:654] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:37.572180 5427 log.cc:824] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:37.572180 5428 log.cc:824] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:37.572405 5429 tablet_bootstrap.cc:654] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:37.573658 5429 log.cc:824] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:37.575467 5427 tablet_bootstrap.cc:492] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: No bootstrap required, opened a new log
I20260927 21:39:37.575534 5428 tablet_bootstrap.cc:492] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: No bootstrap required, opened a new log
I20260927 21:39:37.575759 5427 ts_tablet_manager.cc:1397] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260927 21:39:37.575800 5428 ts_tablet_manager.cc:1397] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 21:39:37.576972 5429 tablet_bootstrap.cc:492] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: No bootstrap required, opened a new log
I20260927 21:39:37.577230 5429 ts_tablet_manager.cc:1397] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260927 21:39:37.583036 5427 raft_consensus.cc:348] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.583436 5427 raft_consensus.cc:374] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:37.583527 5427 raft_consensus.cc:729] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a0d89415dd41d0928b24e1eb0acf1c, State: Initialized, Role: FOLLOWER
I20260927 21:39:37.584127 5427 consensus_queue.cc:260] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.585307 5409 heartbeater.cc:503] Master 127.0.202.190:44017 was elected leader, sending a full tablet report...
I20260927 21:39:37.585903 5427 ts_tablet_manager.cc:1428] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 21:39:37.586493 5428 raft_consensus.cc:348] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.586928 5428 raft_consensus.cc:374] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:37.587034 5428 raft_consensus.cc:729] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9907c8c5c248e7b949cf03e37127c5, State: Initialized, Role: FOLLOWER
I20260927 21:39:37.587639 5428 consensus_queue.cc:260] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.587591 5429 raft_consensus.cc:348] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.587972 5429 raft_consensus.cc:374] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:37.588085 5429 raft_consensus.cc:729] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5158b22d100471aadcff34ae7bcc193, State: Initialized, Role: FOLLOWER
I20260927 21:39:37.588783 5429 consensus_queue.cc:260] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.589296 5276 heartbeater.cc:503] Master 127.0.202.190:44017 was elected leader, sending a full tablet report...
I20260927 21:39:37.590101 5428 ts_tablet_manager.cc:1428] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 21:39:37.591658 5429 ts_tablet_manager.cc:1428] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260927 21:39:37.658087 5144 tablet.cc:2369] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:39:37.696161 5277 tablet.cc:2369] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:37.696749 5435 raft_consensus.cc:482] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:37.697047 5435 raft_consensus.cc:504] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.698848 5435 leader_election.cc:288] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9a0d89415dd41d0928b24e1eb0acf1c (127.0.202.131:36845), 0c9907c8c5c248e7b949cf03e37127c5 (127.0.202.130:41447)
I20260927 21:39:37.707448 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.699678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:56681 (local address 127.0.202.130:41447)
0927 21:39:37.699849 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.699855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.700255 (+ 400us) server_negotiation.cc:407] Connection header received
0927 21:39:37.700454 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.700460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.700554 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.700659 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:37.701426 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.702348 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.704860 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.706281 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.706535 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.706618 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.706948 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.707081 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:39:37.707526 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.699207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:41447 (local address 127.0.202.129:56681)
0927 21:39:37.700051 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:37.700084 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:37.700267 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:37.700794 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:37.700799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:37.700831 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:37.701256 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.701272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.702519 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.702525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.704657 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.704672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.706546 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.706552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.706748 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.706804 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:39:37.706919 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:39:37.707078 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":20,"thread_start_us":542,"threads_started":1}
I20260927 21:39:37.707638 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.699334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:36845 (local address 127.0.202.129:50643)
0927 21:39:37.700528 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:37.700548 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:37.700692 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:37.701087 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:37.701091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:37.701113 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:37.701620 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.701632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.702838 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.702842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.705025 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:37.705042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.706715 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.706721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:37.706931 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.706956 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:39:37.707067 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:39:37.707199 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":5,"thread_start_us":674,"threads_started":1}
I20260927 21:39:37.707500 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.699242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:50643 (local address 127.0.202.131:36845)
0927 21:39:37.699476 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.699485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.700663 (+ 1178us) server_negotiation.cc:407] Connection header received
0927 21:39:37.700853 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.700861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.700947 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.701058 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:37.701800 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.702660 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.705277 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.706326 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.706699 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.706779 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.707109 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.707247 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:39:37.708580 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f810d3150801427ca8b8111e123c9fc3" candidate_uuid: "e5158b22d100471aadcff34ae7bcc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9907c8c5c248e7b949cf03e37127c5" is_pre_election: true
I20260927 21:39:37.708874 5364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f810d3150801427ca8b8111e123c9fc3" candidate_uuid: "e5158b22d100471aadcff34ae7bcc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" is_pre_election: true
I20260927 21:39:37.709110 5231 raft_consensus.cc:2454] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5158b22d100471aadcff34ae7bcc193 in term 0.
I20260927 21:39:37.709388 5364 raft_consensus.cc:2454] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5158b22d100471aadcff34ae7bcc193 in term 0.
I20260927 21:39:37.709910 5034 leader_election.cc:302] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9907c8c5c248e7b949cf03e37127c5, e5158b22d100471aadcff34ae7bcc193; no voters:
I20260927 21:39:37.710306 5435 raft_consensus.cc:2781] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:37.710454 5435 raft_consensus.cc:482] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:37.710556 5435 raft_consensus.cc:3037] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:39:37.711136 5410 tablet.cc:2369] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:37.712831 5435 raft_consensus.cc:504] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.713501 5435 leader_election.cc:288] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [CANDIDATE]: Term 1 election: Requested vote from peers b9a0d89415dd41d0928b24e1eb0acf1c (127.0.202.131:36845), 0c9907c8c5c248e7b949cf03e37127c5 (127.0.202.130:41447)
I20260927 21:39:37.714067 5364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f810d3150801427ca8b8111e123c9fc3" candidate_uuid: "e5158b22d100471aadcff34ae7bcc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c"
I20260927 21:39:37.714036 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f810d3150801427ca8b8111e123c9fc3" candidate_uuid: "e5158b22d100471aadcff34ae7bcc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9907c8c5c248e7b949cf03e37127c5"
I20260927 21:39:37.714277 5364 raft_consensus.cc:3037] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:37.714284 5231 raft_consensus.cc:3037] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:37.717465 5364 raft_consensus.cc:2454] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5158b22d100471aadcff34ae7bcc193 in term 1.
I20260927 21:39:37.717465 5231 raft_consensus.cc:2454] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5158b22d100471aadcff34ae7bcc193 in term 1.
I20260927 21:39:37.717908 5034 leader_election.cc:302] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9907c8c5c248e7b949cf03e37127c5, e5158b22d100471aadcff34ae7bcc193; no voters:
I20260927 21:39:37.718225 5435 raft_consensus.cc:2781] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:37.718700 5435 raft_consensus.cc:686] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 1 LEADER]: Becoming Leader. State: Replica: e5158b22d100471aadcff34ae7bcc193, State: Running, Role: LEADER
I20260927 21:39:37.719278 5435 consensus_queue.cc:237] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } }
I20260927 21:39:37.724268 4953 catalog_manager.cc:5495] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5158b22d100471aadcff34ae7bcc193 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "e5158b22d100471aadcff34ae7bcc193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5158b22d100471aadcff34ae7bcc193" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 42297 } health_report { overall_health: HEALTHY } } }
I20260927 21:39:37.745685 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.740027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.0.202.129:42297)
0927 21:39:37.740221 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.740225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.740242 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:39:37.740387 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.740390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.740465 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.740592 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:37.741000 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.741776 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.743187 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.743730 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.743837 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.744529 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:37.745062 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:37.745254 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.745372 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.745489 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:39:37.754035 5364 raft_consensus.cc:1261] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 1 FOLLOWER]: Refusing update from remote peer e5158b22d100471aadcff34ae7bcc193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:37.754035 5231 raft_consensus.cc:1261] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 1 FOLLOWER]: Refusing update from remote peer e5158b22d100471aadcff34ae7bcc193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:37.755172 5438 consensus_queue.cc:1035] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9907c8c5c248e7b949cf03e37127c5" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:37.755707 5435 consensus_queue.cc:1035] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9a0d89415dd41d0928b24e1eb0acf1c" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:37.782732 4954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:37.783071 4954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:37.787663 4954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19bc45cea4aa404887f36aece65919f2: Sending DeleteTablet for 3 replicas of tablet f810d3150801427ca8b8111e123c9fc3
I20260927 21:39:37.788959 5344 tablet_service.cc:1514] Processing DeleteTablet for tablet f810d3150801427ca8b8111e123c9fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:37 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:58144
I20260927 21:39:37.789410 5078 tablet_service.cc:1514] Processing DeleteTablet for tablet f810d3150801427ca8b8111e123c9fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:37 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:50620
I20260927 21:39:37.790483 5211 tablet_service.cc:1514] Processing DeleteTablet for tablet f810d3150801427ca8b8111e123c9fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:37 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:56140
I20260927 21:39:37.795042 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5016
I20260927 21:39:37.814867 5453 tablet_replica.cc:330] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: stopping tablet replica
I20260927 21:39:37.815825 5453 raft_consensus.cc:2229] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:37.815894 5451 tablet_replica.cc:330] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: stopping tablet replica
I20260927 21:39:37.816370 5451 raft_consensus.cc:2229] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:37.816488 5453 raft_consensus.cc:2258] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:37.816892 5451 raft_consensus.cc:2258] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:37.819032 5453 ts_tablet_manager.cc:1905] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:37.819032 5451 ts_tablet_manager.cc:1905] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:37.816982 5452 tablet_replica.cc:330] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: stopping tablet replica
I20260927 21:39:37.820883 5452 raft_consensus.cc:2229] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:37.821620 5452 raft_consensus.cc:2258] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:37.826114 5452 ts_tablet_manager.cc:1905] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:37.827530 5451 ts_tablet_manager.cc:1918] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:37.827530 5453 ts_tablet_manager.cc:1918] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:37.827721 5453 log.cc:1196] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/f810d3150801427ca8b8111e123c9fc3
I20260927 21:39:37.827721 5451 log.cc:1196] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/f810d3150801427ca8b8111e123c9fc3
I20260927 21:39:37.828336 5451 ts_tablet_manager.cc:1939] T f810d3150801427ca8b8111e123c9fc3 P b9a0d89415dd41d0928b24e1eb0acf1c: Deleting consensus metadata
I20260927 21:39:37.828327 5453 ts_tablet_manager.cc:1939] T f810d3150801427ca8b8111e123c9fc3 P 0c9907c8c5c248e7b949cf03e37127c5: Deleting consensus metadata
I20260927 21:39:37.830332 4942 catalog_manager.cc:4841] TS 0c9907c8c5c248e7b949cf03e37127c5 (127.0.202.130:41447): tablet f810d3150801427ca8b8111e123c9fc3 (table test-table [id=5bfe1d19063a424e8ed2c76024568e5f]) successfully deleted
I20260927 21:39:37.830451 4941 catalog_manager.cc:4841] TS b9a0d89415dd41d0928b24e1eb0acf1c (127.0.202.131:36845): tablet f810d3150801427ca8b8111e123c9fc3 (table test-table [id=5bfe1d19063a424e8ed2c76024568e5f]) successfully deleted
Sep 27 21:39:37 dist-test-slave-25dq krb5kdc[4911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:37.838361 5452 ts_tablet_manager.cc:1918] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:37.838586 5452 log.cc:1196] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/f810d3150801427ca8b8111e123c9fc3
I20260927 21:39:37.839195 5452 ts_tablet_manager.cc:1939] T f810d3150801427ca8b8111e123c9fc3 P e5158b22d100471aadcff34ae7bcc193: Deleting consensus metadata
I20260927 21:39:37.841203 4942 catalog_manager.cc:4841] TS e5158b22d100471aadcff34ae7bcc193 (127.0.202.129:42297): tablet f810d3150801427ca8b8111e123c9fc3 (table test-table [id=5bfe1d19063a424e8ed2c76024568e5f]) successfully deleted
I20260927 21:39:37.843370 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.796726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.0.202.129:42297)
0927 21:39:37.796980 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.796990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.814808 (+ 17818us) server_negotiation.cc:407] Connection header received
0927 21:39:37.815182 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.815192 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.815441 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.815579 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:37.817788 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.819258 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.823518 (+ 4260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.824384 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.824494 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.834860 (+ 10366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:37.834875 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:37.834895 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:37.834976 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:37.839844 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.840847 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.840857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.840860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.841078 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.841545 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.841551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.841555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.842054 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:37.842443 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.842624 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.842910 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:39:37.844172 5138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:37.948973 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5016
I20260927 21:39:37.969798 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5149
Sep 27 21:39:37 dist-test-slave-25dq krb5kdc[4911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:39:37.985522 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:37.970564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.0.202.130:41447)
0927 21:39:37.970772 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:37.970777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:37.970797 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:39:37.970939 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:37.970944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:37.971016 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:37.971112 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:37.972399 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.973137 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.974410 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:37.974887 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:37.974996 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:37.979081 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:37.979094 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:37.979111 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:37.979154 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:37.982490 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.983229 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.983236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.983239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.983441 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:37.983791 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:37.983794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:37.983796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:37.984170 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:37.984572 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:37.984950 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 21:39:37.985126 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 21:39:37.986281 5271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:38.077829 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5149
I20260927 21:39:38.094517 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5282
Sep 27 21:39:38 dist-test-slave-25dq krb5kdc[4911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:39:38.110383 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:38.095425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.202.131:36845)
0927 21:39:38.095610 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:38.095615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:38.095635 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:39:38.095810 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:38.095814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:38.095896 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:38.095993 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:38.097473 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:38.098256 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:38.099568 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:38.099948 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:38.100062 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:38.104763 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:38.104778 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:38.104794 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:38.104863 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:38.107656 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:38.108364 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:38.108372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:38.108374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:38.108571 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:38.108969 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:38.108974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:38.108976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:38.109354 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:38.109625 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:38.109869 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 21:39:38.110045 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:39:38.111042 5404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:38.201464 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5282
I20260927 21:39:38.218297 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 4924
I20260927 21:39:38.219476 4985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:38.331462 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 4924
2026-09-27T21:39:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4376 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T21:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:38Z Disabled control of system clock
I20260927 21:39:38.398942 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33853
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:39:38.725489 5462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:38.725781 5462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:38.725919 5462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:38.733144 5462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:38.733258 5462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:38.733295 5462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:38.733390 5462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:38.733457 5462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:38.742622 5462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:38.743963 5462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:38.745875 5462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:38.757653 5467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:38.757910 5470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:38.757934 5468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:38.759055 5462 server_base.cc:1034] running on GCE node
I20260927 21:39:38.759931 5462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:38.761453 5462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:38.762693 5462 hybrid_clock.cc:648] HybridClock initialized: now 1790545178762646 us; error 39 us; skew 500 ppm
I20260927 21:39:38.763049 5462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:38.776080 5462 webserver.cc:460] Webserver started at http://127.0.202.190:33929/ using document root <none> and password file <none>
I20260927 21:39:38.776969 5462 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:38.777101 5462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:38.777431 5462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:38.780450 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "b03f21ce97b2487a8469988ebe2f1347"
format_stamp: "Formatted at 2026-09-27 21:39:38 on dist-test-slave-25dq"
server_key: "0eefd26ebd7b34bc9c66cc0e33d02c20"
server_key_iv: "cc1098257efb4ea19fb5784bb92e5965"
server_key_version: "encryptionkey@0"
I20260927 21:39:38.781242 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "b03f21ce97b2487a8469988ebe2f1347"
format_stamp: "Formatted at 2026-09-27 21:39:38 on dist-test-slave-25dq"
server_key: "0eefd26ebd7b34bc9c66cc0e33d02c20"
server_key_iv: "cc1098257efb4ea19fb5784bb92e5965"
server_key_version: "encryptionkey@0"
I20260927 21:39:38.786579 5462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:39:38.790063 5476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:38.791345 5462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:39:38.791524 5462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "b03f21ce97b2487a8469988ebe2f1347"
format_stamp: "Formatted at 2026-09-27 21:39:38 on dist-test-slave-25dq"
server_key: "0eefd26ebd7b34bc9c66cc0e33d02c20"
server_key_iv: "cc1098257efb4ea19fb5784bb92e5965"
server_key_version: "encryptionkey@0"
I20260927 21:39:38.791769 5462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:38.797781 5462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:38.798583 5462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:38.798873 5462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:38.816944 5462 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:33853
I20260927 21:39:38.817013 5527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:33853 every 8 connection(s)
I20260927 21:39:38.818310 5462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:38.822305 5528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:38.828318 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5462
I20260927 21:39:38.828514 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:38.829339 810 external_mini_cluster.cc:1428] Setting key 24c5f84497511e96b64ce62419fa060a
I20260927 21:39:38.834821 5528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347: Bootstrap starting.
I20260927 21:39:38.838102 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:38.831617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.0.202.190:33853)
0927 21:39:38.832755 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:38.832767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:38.832808 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:39:38.834411 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:38.834437 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:38.834856 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:38.835225 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:38.836496 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:38.836502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:38.836514 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:38.836532 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:38.836594 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:38.836613 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:38.836779 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:38.836924 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 21:39:38.837167 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":421,"threads_started":1}
I20260927 21:39:38.838709 5528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:38.839793 5528 log.cc:824] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:38.842818 5528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347: No bootstrap required, opened a new log
I20260927 21:39:38.850916 5528 raft_consensus.cc:348] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f21ce97b2487a8469988ebe2f1347" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33853 } }
I20260927 21:39:38.851352 5528 raft_consensus.cc:374] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:38.851477 5528 raft_consensus.cc:729] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03f21ce97b2487a8469988ebe2f1347, State: Initialized, Role: FOLLOWER
I20260927 21:39:38.852078 5528 consensus_queue.cc:260] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f21ce97b2487a8469988ebe2f1347" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33853 } }
I20260927 21:39:38.852394 5528 raft_consensus.cc:388] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:38.852552 5528 raft_consensus.cc:482] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:38.852734 5528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:38.855093 5528 raft_consensus.cc:504] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f21ce97b2487a8469988ebe2f1347" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33853 } }
I20260927 21:39:38.855734 5528 leader_election.cc:302] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b03f21ce97b2487a8469988ebe2f1347; no voters:
I20260927 21:39:38.856426 5528 leader_election.cc:288] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:38.856559 5533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:38.857292 5533 raft_consensus.cc:686] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [term 1 LEADER]: Becoming Leader. State: Replica: b03f21ce97b2487a8469988ebe2f1347, State: Running, Role: LEADER
I20260927 21:39:38.857966 5533 consensus_queue.cc:237] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f21ce97b2487a8469988ebe2f1347" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33853 } }
I20260927 21:39:38.858659 5528 sys_catalog.cc:564] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:38.861105 5534 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b03f21ce97b2487a8469988ebe2f1347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f21ce97b2487a8469988ebe2f1347" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33853 } } }
I20260927 21:39:38.861526 5534 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:38.862349 5535 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03f21ce97b2487a8469988ebe2f1347. Latest consensus state: current_term: 1 leader_uuid: "b03f21ce97b2487a8469988ebe2f1347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f21ce97b2487a8469988ebe2f1347" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33853 } } }
I20260927 21:39:38.862735 5535 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:38.864794 5542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:38.869284 5542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:38.878286 5542 catalog_manager.cc:1347] Generated new cluster ID: a148305b56104ae781ea7942343e9586
I20260927 21:39:38.878430 5542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:38.888278 5542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:38.888770 5542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:38.896829 5542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347: Generated new TSK 0
I20260927 21:39:38.897949 5542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:39:38.912739 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:33853
--builtin_ntp_servers=127.0.202.148:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:39:39.259298 5552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:39.259847 5552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:39.260015 5552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:39.267645 5552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:39.267753 5552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:39.267887 5552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:39.277354 5552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:33853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:39.278631 5552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:39.280576 5552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:39.296167 5560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:39.296252 5558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:39.296226 5557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:39.297079 5552 server_base.cc:1034] running on GCE node
I20260927 21:39:39.297714 5552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:39.298718 5552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:39.299942 5552 hybrid_clock.cc:648] HybridClock initialized: now 1790545179299890 us; error 47 us; skew 500 ppm
I20260927 21:39:39.300276 5552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:39.302811 5552 webserver.cc:460] Webserver started at http://127.0.202.129:36967/ using document root <none> and password file <none>
I20260927 21:39:39.303565 5552 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:39.303692 5552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:39.304013 5552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:39.306902 5552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "d324cc6155ad4e0a9274bc9519fb2b11"
format_stamp: "Formatted at 2026-09-27 21:39:39 on dist-test-slave-25dq"
server_key: "56e9e57304a0cdb99b394e725e2f2963"
server_key_iv: "d522a36a8c8ed36edc25c206208d52c7"
server_key_version: "encryptionkey@0"
I20260927 21:39:39.307713 5552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "d324cc6155ad4e0a9274bc9519fb2b11"
format_stamp: "Formatted at 2026-09-27 21:39:39 on dist-test-slave-25dq"
server_key: "56e9e57304a0cdb99b394e725e2f2963"
server_key_iv: "d522a36a8c8ed36edc25c206208d52c7"
server_key_version: "encryptionkey@0"
I20260927 21:39:39.312964 5552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:39:39.316733 5566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:39.318073 5552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:39:39.318248 5552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "d324cc6155ad4e0a9274bc9519fb2b11"
format_stamp: "Formatted at 2026-09-27 21:39:39 on dist-test-slave-25dq"
server_key: "56e9e57304a0cdb99b394e725e2f2963"
server_key_iv: "d522a36a8c8ed36edc25c206208d52c7"
server_key_version: "encryptionkey@0"
I20260927 21:39:39.318476 5552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:39.324208 5552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:39.325071 5552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:39.325346 5552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:39.326292 5552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:39.327764 5552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:39.327881 5552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:39.328006 5552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:39.328087 5552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:39.362716 5552 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:33189
I20260927 21:39:39.362772 5678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:33189 every 8 connection(s)
I20260927 21:39:39.364094 5552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:39.372795 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:39.366923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:53841 (local address 127.0.202.190:33853)
0927 21:39:39.367369 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:39.367374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:39.368755 (+ 1381us) server_negotiation.cc:407] Connection header received
0927 21:39:39.370600 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:39.370605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:39.370663 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:39.370776 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:39.372079 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:39.372083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:39.372086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:39.372102 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:39.372139 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:39.372159 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:39.372272 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:39.372523 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 21:39:39.372575 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260927 21:39:39.373809 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:39.367647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33853 (local address 127.0.202.129:53841)
0927 21:39:39.368604 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:39.368642 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:39.370168 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:39.370986 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:39.371008 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:39.371105 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:39.371770 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:39.371772 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:39.371810 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:39.371907 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:39.372258 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:39.372265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:39.372271 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:39:39.372553 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 21:39:39.372823 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":253,"threads_started":1}
I20260927 21:39:39.374429 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5552
I20260927 21:39:39.374689 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:39.375398 810 external_mini_cluster.cc:1428] Setting key 7cc3cf592e8ae793b113645874050349
I20260927 21:39:39.375492 5679 heartbeater.cc:347] Connected to a master server at 127.0.202.190:33853
I20260927 21:39:39.375785 5679 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:39.376570 5679 heartbeater.cc:511] Master 127.0.202.190:33853 requested a full tablet report, sending...
I20260927 21:39:39.378509 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:33853
--builtin_ntp_servers=127.0.202.148:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:39:39.379037 5493 ts_manager.cc:194] Registered new tserver with Master: d324cc6155ad4e0a9274bc9519fb2b11 (127.0.202.129:33189)
W20260927 21:39:39.705754 5683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:39.706056 5683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:39.706203 5683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:39.713696 5683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:39.713809 5683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:39.713987 5683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:39.723569 5683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:33853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:39.724953 5683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:39.726759 5683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:39.741006 5688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:39.741271 5691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:39.741261 5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:39.742672 5683 server_base.cc:1034] running on GCE node
I20260927 21:39:39.743435 5683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:39.744575 5683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:39.745823 5683 hybrid_clock.cc:648] HybridClock initialized: now 1790545179745746 us; error 68 us; skew 500 ppm
I20260927 21:39:39.746176 5683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:39.748963 5683 webserver.cc:460] Webserver started at http://127.0.202.130:39345/ using document root <none> and password file <none>
I20260927 21:39:39.749727 5683 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:39.749861 5683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:39.750213 5683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:39.753477 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af"
format_stamp: "Formatted at 2026-09-27 21:39:39 on dist-test-slave-25dq"
server_key: "31ea1b113eef5c9388297ac17da64ac1"
server_key_iv: "536a54482ec0a72f5f63f2b903eb43eb"
server_key_version: "encryptionkey@0"
I20260927 21:39:39.754369 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af"
format_stamp: "Formatted at 2026-09-27 21:39:39 on dist-test-slave-25dq"
server_key: "31ea1b113eef5c9388297ac17da64ac1"
server_key_iv: "536a54482ec0a72f5f63f2b903eb43eb"
server_key_version: "encryptionkey@0"
I20260927 21:39:39.760543 5683 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 21:39:39.764776 5697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:39.766376 5683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:39.766600 5683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af"
format_stamp: "Formatted at 2026-09-27 21:39:39 on dist-test-slave-25dq"
server_key: "31ea1b113eef5c9388297ac17da64ac1"
server_key_iv: "536a54482ec0a72f5f63f2b903eb43eb"
server_key_version: "encryptionkey@0"
I20260927 21:39:39.766847 5683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:39.773002 5683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:39.773818 5683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:39.774089 5683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:39.774984 5683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:39.776491 5683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:39.776608 5683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:39.776734 5683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:39.776818 5683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:39.809574 5683 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:42931
I20260927 21:39:39.809749 5809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:42931 every 8 connection(s)
I20260927 21:39:39.810956 5683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:39.819367 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5683
I20260927 21:39:39.819589 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:39.819855 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:39.813579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:41709 (local address 127.0.202.190:33853)
0927 21:39:39.813774 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:39.813778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:39.815222 (+ 1444us) server_negotiation.cc:407] Connection header received
0927 21:39:39.817260 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:39.817264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:39.817315 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:39.817476 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:39.819012 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:39.819020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:39.819027 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:39.819052 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:39.819095 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:39.819116 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:39.819267 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:39.819462 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 21:39:39.819536 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:39:39.820174 810 external_mini_cluster.cc:1428] Setting key 1bc0313b14c576b9a20350eb578c60eb
I20260927 21:39:39.820959 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:39.814210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33853 (local address 127.0.202.130:41709)
0927 21:39:39.815094 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:39.815137 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:39.816952 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:39.817658 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:39.817669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:39.817790 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:39.818620 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:39.818623 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:39.818674 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:39.818783 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:39.819228 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:39.819237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:39.819245 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:39:39.819496 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 21:39:39.819755 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":260,"threads_started":1}
I20260927 21:39:39.822408 5810 heartbeater.cc:347] Connected to a master server at 127.0.202.190:33853
I20260927 21:39:39.822722 5810 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:39.823421 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:33853
--builtin_ntp_servers=127.0.202.148:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:39:39.823482 5810 heartbeater.cc:511] Master 127.0.202.190:33853 requested a full tablet report, sending...
I20260927 21:39:39.825026 5493 ts_manager.cc:194] Registered new tserver with Master: af8d8b1fe9a94948a6baddd3e0f3d2af (127.0.202.130:42931)
W20260927 21:39:40.156975 5813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:40.157312 5813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:40.157465 5813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:40.164887 5813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:40.165033 5813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:40.165205 5813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:40.174588 5813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:33853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:40.175909 5813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:40.177770 5813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:40.191116 5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:40.191149 5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:40.191214 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:40.191890 5813 server_base.cc:1034] running on GCE node
I20260927 21:39:40.192585 5813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:40.193610 5813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:40.194852 5813 hybrid_clock.cc:648] HybridClock initialized: now 1790545180194764 us; error 82 us; skew 500 ppm
I20260927 21:39:40.195226 5813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:40.197800 5813 webserver.cc:460] Webserver started at http://127.0.202.131:43887/ using document root <none> and password file <none>
I20260927 21:39:40.198522 5813 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:40.198647 5813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:40.198959 5813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:40.201957 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "1a347b1fe203466d99ded2dcf5d43a74"
format_stamp: "Formatted at 2026-09-27 21:39:40 on dist-test-slave-25dq"
server_key: "3f08489ef852f2326d7ff75bd2b06531"
server_key_iv: "002741b3e012f6c9b7e9d72aeaaa31eb"
server_key_version: "encryptionkey@0"
I20260927 21:39:40.202787 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "1a347b1fe203466d99ded2dcf5d43a74"
format_stamp: "Formatted at 2026-09-27 21:39:40 on dist-test-slave-25dq"
server_key: "3f08489ef852f2326d7ff75bd2b06531"
server_key_iv: "002741b3e012f6c9b7e9d72aeaaa31eb"
server_key_version: "encryptionkey@0"
I20260927 21:39:40.208366 5813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:39:40.212108 5827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:40.213712 5813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:39:40.213902 5813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "1a347b1fe203466d99ded2dcf5d43a74"
format_stamp: "Formatted at 2026-09-27 21:39:40 on dist-test-slave-25dq"
server_key: "3f08489ef852f2326d7ff75bd2b06531"
server_key_iv: "002741b3e012f6c9b7e9d72aeaaa31eb"
server_key_version: "encryptionkey@0"
I20260927 21:39:40.214136 5813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:40.219823 5813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:40.220645 5813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:40.220916 5813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:40.221846 5813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:40.223443 5813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:40.223547 5813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:40.223621 5813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:40.223667 5813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:40.257671 5813 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:41625
I20260927 21:39:40.257779 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:41625 every 8 connection(s)
I20260927 21:39:40.259063 5813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:40.265745 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5813
I20260927 21:39:40.265921 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:40.266494 810 external_mini_cluster.cc:1428] Setting key 152262b4d278d8184755dd71f89a4f1b
I20260927 21:39:40.268457 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.262211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:45417 (local address 127.0.202.190:33853)
0927 21:39:40.262417 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.262423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.264117 (+ 1694us) server_negotiation.cc:407] Connection header received
0927 21:39:40.266033 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.266037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.266090 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.266180 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.267727 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.267731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.267735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.267749 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.267777 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.267787 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.267908 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.268104 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.268170 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 21:39:40.269594 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.262894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33853 (local address 127.0.202.131:45417)
0927 21:39:40.263921 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.263968 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:40.265710 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:40.266442 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:40.266454 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:40.266597 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:40.267373 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:40.267375 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:40.267434 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:40.267555 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:40.267878 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:40.267889 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:40.267897 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:39:40.268188 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 21:39:40.268542 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":279,"threads_started":1}
I20260927 21:39:40.271087 5940 heartbeater.cc:347] Connected to a master server at 127.0.202.190:33853
I20260927 21:39:40.271394 5940 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:40.272014 5940 heartbeater.cc:511] Master 127.0.202.190:33853 requested a full tablet report, sending...
I20260927 21:39:40.273440 5493 ts_manager.cc:194] Registered new tserver with Master: 1a347b1fe203466d99ded2dcf5d43a74 (127.0.202.131:41625)
I20260927 21:39:40.281864 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:39:40.288955 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.286344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.0.202.190:33853)
0927 21:39:40.286515 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.286519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.286987 (+ 468us) server_negotiation.cc:407] Connection header received
0927 21:39:40.287143 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.287147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.287215 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.287325 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.288288 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.288337 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.288345 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.288416 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.288446 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.288459 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.288563 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.288697 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.288742 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:39:40.293275 5493 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:40.300047 5493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:40.322086 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.312361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.202.131:41625)
0927 21:39:40.318298 (+ 5937us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.318306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.318341 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:39:40.318468 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.318477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.318726 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.319525 (+ 799us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.320988 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.320996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.321012 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.321029 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.321090 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.321112 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.321547 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.321619 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.321718 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5768,"thread_start_us":5630,"threads_started":1}
I20260927 21:39:40.322122 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.312159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41625 (local address 127.0.0.1:54636)
0927 21:39:40.316912 (+ 4753us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.316955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:40.317098 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:40.318932 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:40.318940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:40.319033 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:40.319931 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:40.319935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:40.319963 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:40.320037 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:40.321227 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:40.321238 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:40.321246 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:39:40.321523 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 21:39:40.321628 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4507,"thread_start_us":5705,"threads_started":1}
I20260927 21:39:40.324935 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.312741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.0.202.129:33189)
0927 21:39:40.313247 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.313256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.318393 (+ 5137us) server_negotiation.cc:407] Connection header received
0927 21:39:40.318618 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.318625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.318786 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.318906 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.323967 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.323978 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.323993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.324022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.324085 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.324108 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.324223 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.324510 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.324622 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":430,"threads_started":1}
I20260927 21:39:40.324975 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.312978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41268)
0927 21:39:40.318200 (+ 5222us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.318218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:40.318475 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:40.319969 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:40.319973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:40.319994 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:40.323457 (+ 3463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:40.323469 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:40.323488 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:40.323533 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:40.324343 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:40.324350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:40.324355 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:39:40.324504 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 21:39:40.324554 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5079,"thread_start_us":3521,"threads_started":1}
I20260927 21:39:40.325910 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.318441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:42931 (local address 127.0.0.1:42240)
0927 21:39:40.322313 (+ 3872us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.322330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:40.322411 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:40.323450 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:40.323454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:40.323494 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:40.323993 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:40.323996 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:40.324010 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:40.324051 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:40.325581 (+ 1530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:40.325587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:40.325594 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:39:40.325668 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:39:40.325731 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3770,"thread_start_us":4429,"threads_started":1}
I20260927 21:39:40.326686 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.318709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.0.202.130:42931)
0927 21:39:40.323041 (+ 4332us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.323049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.323073 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:39:40.323125 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.323131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.323295 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.323453 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.324325 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.324335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.324348 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.324370 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.324435 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.324458 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.326233 (+ 1775us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.326310 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.326400 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4201,"thread_start_us":4078,"threads_started":1}
I20260927 21:39:40.327188 5875 tablet_service.cc:1467] Processing CreateTablet for tablet 2001efb665ef445ca9903c513bd8c7e6 (DEFAULT_TABLE table=test-table [id=086427c9646e4bdd83d66c2966a208d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:40.328722 5875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2001efb665ef445ca9903c513bd8c7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:40.328670 5614 tablet_service.cc:1467] Processing CreateTablet for tablet 2001efb665ef445ca9903c513bd8c7e6 (DEFAULT_TABLE table=test-table [id=086427c9646e4bdd83d66c2966a208d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:40.329797 5614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2001efb665ef445ca9903c513bd8c7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:40.330106 5745 tablet_service.cc:1467] Processing CreateTablet for tablet 2001efb665ef445ca9903c513bd8c7e6 (DEFAULT_TABLE table=test-table [id=086427c9646e4bdd83d66c2966a208d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:40.331670 5745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2001efb665ef445ca9903c513bd8c7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:40.340763 5958 tablet_bootstrap.cc:492] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11: Bootstrap starting.
I20260927 21:39:40.344534 5958 tablet_bootstrap.cc:654] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:40.345566 5958 log.cc:824] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:40.348111 5961 tablet_bootstrap.cc:492] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Bootstrap starting.
I20260927 21:39:40.348760 5958 tablet_bootstrap.cc:492] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11: No bootstrap required, opened a new log
I20260927 21:39:40.349095 5958 ts_tablet_manager.cc:1397] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260927 21:39:40.349125 5959 tablet_bootstrap.cc:492] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Bootstrap starting.
I20260927 21:39:40.352213 5961 tablet_bootstrap.cc:654] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:40.352941 5959 tablet_bootstrap.cc:654] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:40.353610 5961 log.cc:824] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:40.354192 5959 log.cc:824] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:40.356968 5961 tablet_bootstrap.cc:492] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: No bootstrap required, opened a new log
I20260927 21:39:40.357254 5961 ts_tablet_manager.cc:1397] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 21:39:40.357349 5959 tablet_bootstrap.cc:492] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: No bootstrap required, opened a new log
I20260927 21:39:40.357627 5959 ts_tablet_manager.cc:1397] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260927 21:39:40.358247 5958 raft_consensus.cc:348] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.358711 5958 raft_consensus.cc:374] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:40.358814 5958 raft_consensus.cc:729] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d324cc6155ad4e0a9274bc9519fb2b11, State: Initialized, Role: FOLLOWER
I20260927 21:39:40.359565 5958 consensus_queue.cc:260] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.360802 5679 heartbeater.cc:503] Master 127.0.202.190:33853 was elected leader, sending a full tablet report...
I20260927 21:39:40.361423 5958 ts_tablet_manager.cc:1428] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260927 21:39:40.366515 5680 tablet.cc:2369] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:40.368882 5959 raft_consensus.cc:348] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.368808 5961 raft_consensus.cc:348] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.369314 5961 raft_consensus.cc:374] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:40.369424 5959 raft_consensus.cc:374] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:40.369437 5961 raft_consensus.cc:729] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8d8b1fe9a94948a6baddd3e0f3d2af, State: Initialized, Role: FOLLOWER
I20260927 21:39:40.369589 5959 raft_consensus.cc:729] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a347b1fe203466d99ded2dcf5d43a74, State: Initialized, Role: FOLLOWER
I20260927 21:39:40.370136 5961 consensus_queue.cc:260] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.370224 5959 consensus_queue.cc:260] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.371359 5810 heartbeater.cc:503] Master 127.0.202.190:33853 was elected leader, sending a full tablet report...
I20260927 21:39:40.372200 5961 ts_tablet_manager.cc:1428] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 21:39:40.373263 5940 heartbeater.cc:503] Master 127.0.202.190:33853 was elected leader, sending a full tablet report...
I20260927 21:39:40.375082 5959 ts_tablet_manager.cc:1428] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260927 21:39:40.421556 5964 raft_consensus.cc:482] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:40.421936 5964 raft_consensus.cc:504] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.423585 5964 leader_election.cc:288] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a347b1fe203466d99ded2dcf5d43a74 (127.0.202.131:41625), af8d8b1fe9a94948a6baddd3e0f3d2af (127.0.202.130:42931)
I20260927 21:39:40.427086 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.424136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:39327 (local address 127.0.202.131:41625)
0927 21:39:40.424337 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.424344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.424793 (+ 449us) server_negotiation.cc:407] Connection header received
0927 21:39:40.424962 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.424968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.425046 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.425159 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.426380 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.426386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.426390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.426413 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.426441 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.426452 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.426576 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.426725 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.426784 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:39:40.427242 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.423994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41625 (local address 127.0.202.129:39327)
0927 21:39:40.424685 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.424707 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:40.424830 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:40.425325 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:40.425332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:40.425368 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:40.426190 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:40.426192 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:40.426205 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:40.426239 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:40.426590 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:40.426597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:40.426605 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:39:40.426750 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 21:39:40.426796 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":351,"threads_started":1}
I20260927 21:39:40.427309 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.425137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:48343 (local address 127.0.202.130:42931)
0927 21:39:40.425314 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.425319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.425460 (+ 141us) server_negotiation.cc:407] Connection header received
0927 21:39:40.425561 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.425569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.425648 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.425757 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.426648 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.426654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.426658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.426676 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.426708 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.426719 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.426832 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.426934 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.427002 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:39:40.427739 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.424812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:42931 (local address 127.0.202.129:48343)
0927 21:39:40.425368 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.425381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:40.425472 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:40.425806 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:40.425813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:40.425836 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:40.426408 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:39:40.426410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:40.426423 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:39:40.426503 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:40.426807 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:40.426811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:40.426816 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:39:40.426975 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 21:39:40.427019 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":295,"threads_started":1}
I20260927 21:39:40.428521 5895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2001efb665ef445ca9903c513bd8c7e6" candidate_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a347b1fe203466d99ded2dcf5d43a74" is_pre_election: true
I20260927 21:39:40.428644 5765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2001efb665ef445ca9903c513bd8c7e6" candidate_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" is_pre_election: true
I20260927 21:39:40.429049 5895 raft_consensus.cc:2454] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d324cc6155ad4e0a9274bc9519fb2b11 in term 0.
I20260927 21:39:40.429183 5765 raft_consensus.cc:2454] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d324cc6155ad4e0a9274bc9519fb2b11 in term 0.
I20260927 21:39:40.429728 5570 leader_election.cc:302] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a347b1fe203466d99ded2dcf5d43a74, d324cc6155ad4e0a9274bc9519fb2b11; no voters:
I20260927 21:39:40.430106 5964 raft_consensus.cc:2781] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:40.430256 5964 raft_consensus.cc:482] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:40.430366 5964 raft_consensus.cc:3037] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:40.432580 5964 raft_consensus.cc:504] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.433138 5964 leader_election.cc:288] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [CANDIDATE]: Term 1 election: Requested vote from peers 1a347b1fe203466d99ded2dcf5d43a74 (127.0.202.131:41625), af8d8b1fe9a94948a6baddd3e0f3d2af (127.0.202.130:42931)
I20260927 21:39:40.433686 5765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2001efb665ef445ca9903c513bd8c7e6" candidate_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af"
I20260927 21:39:40.433701 5895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2001efb665ef445ca9903c513bd8c7e6" candidate_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a347b1fe203466d99ded2dcf5d43a74"
I20260927 21:39:40.433876 5765 raft_consensus.cc:3037] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:40.433916 5895 raft_consensus.cc:3037] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:40.436041 5765 raft_consensus.cc:2454] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d324cc6155ad4e0a9274bc9519fb2b11 in term 1.
I20260927 21:39:40.436043 5895 raft_consensus.cc:2454] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d324cc6155ad4e0a9274bc9519fb2b11 in term 1.
I20260927 21:39:40.436447 5570 leader_election.cc:302] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af8d8b1fe9a94948a6baddd3e0f3d2af, d324cc6155ad4e0a9274bc9519fb2b11; no voters:
I20260927 21:39:40.436739 5964 raft_consensus.cc:2781] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:40.437199 5964 raft_consensus.cc:686] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [term 1 LEADER]: Becoming Leader. State: Replica: d324cc6155ad4e0a9274bc9519fb2b11, State: Running, Role: LEADER
I20260927 21:39:40.437759 5964 consensus_queue.cc:237] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } }
I20260927 21:39:40.442991 5492 catalog_manager.cc:5495] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 reported cstate change: term changed from 0 to 1, leader changed from <none> to d324cc6155ad4e0a9274bc9519fb2b11 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d324cc6155ad4e0a9274bc9519fb2b11" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 33189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 } health_report { overall_health: UNKNOWN } } }
I20260927 21:39:40.483386 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.480868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.0.202.129:33189)
0927 21:39:40.481115 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.481122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.481225 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:39:40.481562 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.481567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.481648 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.481834 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.482776 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.482786 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.482789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.482805 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.482834 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.482846 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.482988 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.483042 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.483099 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 21:39:40.491945 5765 raft_consensus.cc:1261] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 1 FOLLOWER]: Refusing update from remote peer d324cc6155ad4e0a9274bc9519fb2b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:40.491943 5895 raft_consensus.cc:1261] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 1 FOLLOWER]: Refusing update from remote peer d324cc6155ad4e0a9274bc9519fb2b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:40.492902 5964 consensus_queue.cc:1035] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a347b1fe203466d99ded2dcf5d43a74" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:40.493228 5969 consensus_queue.cc:1035] T 2001efb665ef445ca9903c513bd8c7e6 P d324cc6155ad4e0a9274bc9519fb2b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8d8b1fe9a94948a6baddd3e0f3d2af" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:40.520218 5492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:40.520551 5492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:40.524335 5492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b03f21ce97b2487a8469988ebe2f1347: Sending DeleteTablet for 3 replicas of tablet 2001efb665ef445ca9903c513bd8c7e6
I20260927 21:39:40.525563 5614 tablet_service.cc:1514] Processing DeleteTablet for tablet 2001efb665ef445ca9903c513bd8c7e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:40 UTC) from {username='slave'} at 127.0.0.1:41268
I20260927 21:39:40.525633 5745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2001efb665ef445ca9903c513bd8c7e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:40 UTC) from {username='slave'} at 127.0.0.1:42240
I20260927 21:39:40.525635 5875 tablet_service.cc:1514] Processing DeleteTablet for tablet 2001efb665ef445ca9903c513bd8c7e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:40 UTC) from {username='slave'} at 127.0.0.1:54636
I20260927 21:39:40.528126 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5552
I20260927 21:39:40.537981 5984 tablet_replica.cc:330] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: stopping tablet replica
I20260927 21:39:40.538496 5984 raft_consensus.cc:2229] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:40.539037 5984 raft_consensus.cc:2258] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:40.539806 5983 tablet_replica.cc:330] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: stopping tablet replica
I20260927 21:39:40.540256 5983 raft_consensus.cc:2229] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:40.540763 5983 raft_consensus.cc:2258] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:40.544842 5983 ts_tablet_manager.cc:1905] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:40.544924 5984 ts_tablet_manager.cc:1905] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:40.549726 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.534316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.0.202.129:33189)
0927 21:39:40.534534 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.534544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.540876 (+ 6332us) server_negotiation.cc:407] Connection header received
0927 21:39:40.541174 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.541178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.541230 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.541344 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.548348 (+ 7004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.548357 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.548362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.548390 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.548438 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.548458 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.548601 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.549301 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.549377 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:39:40.550998 5674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:40.552781 5983 ts_tablet_manager.cc:1918] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:40.553017 5983 log.cc:1196] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/2001efb665ef445ca9903c513bd8c7e6
I20260927 21:39:40.553260 5984 ts_tablet_manager.cc:1918] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:40.553503 5984 log.cc:1196] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/2001efb665ef445ca9903c513bd8c7e6
I20260927 21:39:40.553660 5983 ts_tablet_manager.cc:1939] T 2001efb665ef445ca9903c513bd8c7e6 P af8d8b1fe9a94948a6baddd3e0f3d2af: Deleting consensus metadata
I20260927 21:39:40.554647 5984 ts_tablet_manager.cc:1939] T 2001efb665ef445ca9903c513bd8c7e6 P 1a347b1fe203466d99ded2dcf5d43a74: Deleting consensus metadata
I20260927 21:39:40.556171 5480 catalog_manager.cc:4841] TS af8d8b1fe9a94948a6baddd3e0f3d2af (127.0.202.130:42931): tablet 2001efb665ef445ca9903c513bd8c7e6 (table test-table [id=086427c9646e4bdd83d66c2966a208d0]) successfully deleted
I20260927 21:39:40.557444 5480 catalog_manager.cc:4841] TS 1a347b1fe203466d99ded2dcf5d43a74 (127.0.202.131:41625): tablet 2001efb665ef445ca9903c513bd8c7e6 (table test-table [id=086427c9646e4bdd83d66c2966a208d0]) successfully deleted
I20260927 21:39:40.661060 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5552
W20260927 21:39:40.679419 5477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.202.129:33189 (error 108)
I20260927 21:39:40.679718 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5683
I20260927 21:39:40.681281 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.680871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41288)
0927 21:39:40.681032 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.681157 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 21:39:40.681484 5477 catalog_manager.cc:4584] TS d324cc6155ad4e0a9274bc9519fb2b11 (127.0.202.129:33189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2001efb665ef445ca9903c513bd8c7e6: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
I20260927 21:39:40.682623 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.680545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.0.202.130:42931)
0927 21:39:40.680754 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.680761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.680782 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:39:40.680939 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.680944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.681027 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.681131 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.682031 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.682036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.682041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.682059 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.682094 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.682105 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.682229 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.682307 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.682365 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:39:40.683328 5805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:40.735584 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.734916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41300)
0927 21:39:40.735176 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.735325 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:39:40.736903 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.736386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41304)
0927 21:39:40.736560 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.736723 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:39:40.788501 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5683
I20260927 21:39:40.797461 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.796885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41314)
0927 21:39:40.797119 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.797241 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:39:40.798635 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.798156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41320)
0927 21:39:40.798346 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.798445 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:39:40.806409 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5813
I20260927 21:39:40.809450 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.807244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.0.202.131:41625)
0927 21:39:40.807398 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:40.807402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:40.807420 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:39:40.807590 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:40.807594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:40.807670 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:40.807757 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:40.808792 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:40.808797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:40.808802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:40.808822 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:40.808868 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:40.808883 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:40.809029 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:40.809097 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:39:40.809155 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:39:40.810175 5935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:40.879384 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.878850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41336)
0927 21:39:40.879096 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.879205 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 21:39:40.880569 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:40.880119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:33189 (local address 127.0.0.1:41348)
0927 21:39:40.880381 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:40.880450 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 21:39:40.900571 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5813
I20260927 21:39:40.917845 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5462
I20260927 21:39:40.919212 5523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:41.025629 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5462
2026-09-27T21:39:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2694 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 21:39:41.067788 810 mini_oidc.cc:152] Starting JWKS server
I20260927 21:39:41.070430 810 webserver.cc:460] Webserver started at https://127.0.0.1:43039/ using document root <none> and password file <none>
I20260927 21:39:41.070758 810 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:39:41.071390 810 webserver.cc:460] Webserver started at http://127.0.0.1:41471/ using document root <none> and password file <none>
2026-09-27T21:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:41Z Disabled control of system clock
I20260927 21:39:41.098526 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:39457
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:39457
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43039/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/testchainca.cert with env {}
W20260927 21:39:41.428579 5996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:41.428870 5996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:41.428956 5996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:41.436483 5996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:41.436601 5996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:41.436638 5996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:41.436687 5996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:41.436729 5996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:39:41.436784 5996 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:39:41.436822 5996 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43039/jwks.json
I20260927 21:39:41.447316 5996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:39457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:39457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43039/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:41.448858 5996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:41.450502 5996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:41.463521 6001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:41.463608 6002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:41.463943 6004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:41.464476 5996 server_base.cc:1034] running on GCE node
I20260927 21:39:41.465340 5996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:41.467002 5996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:41.468263 5996 hybrid_clock.cc:648] HybridClock initialized: now 1790545181468213 us; error 45 us; skew 500 ppm
I20260927 21:39:41.468711 5996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:41.471122 5996 webserver.cc:460] Webserver started at http://127.0.202.190:32893/ using document root <none> and password file <none>
I20260927 21:39:41.471853 5996 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:41.471972 5996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:41.472275 5996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:41.475236 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "e5e8f0d2a8c440b49d58321ae915f26f"
format_stamp: "Formatted at 2026-09-27 21:39:41 on dist-test-slave-25dq"
server_key: "176952057d12efb570e9ecd096fcd6c5"
server_key_iv: "c445281e545306c404ac8d2e90ac8ea4"
server_key_version: "encryptionkey@0"
I20260927 21:39:41.476032 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "e5e8f0d2a8c440b49d58321ae915f26f"
format_stamp: "Formatted at 2026-09-27 21:39:41 on dist-test-slave-25dq"
server_key: "176952057d12efb570e9ecd096fcd6c5"
server_key_iv: "c445281e545306c404ac8d2e90ac8ea4"
server_key_version: "encryptionkey@0"
I20260927 21:39:41.481243 5996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:39:41.484731 6010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:41.486140 5996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:39:41.486340 5996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "e5e8f0d2a8c440b49d58321ae915f26f"
format_stamp: "Formatted at 2026-09-27 21:39:41 on dist-test-slave-25dq"
server_key: "176952057d12efb570e9ecd096fcd6c5"
server_key_iv: "c445281e545306c404ac8d2e90ac8ea4"
server_key_version: "encryptionkey@0"
I20260927 21:39:41.486533 5996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:41.526402 5996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:41.527307 5996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:41.527658 5996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:41.546597 6061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:39457 every 8 connection(s)
I20260927 21:39:41.546590 5996 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:39457
I20260927 21:39:41.548069 5996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:41.551891 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:41.558203 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 5996
I20260927 21:39:41.558439 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:41.559046 810 external_mini_cluster.cc:1428] Setting key 3d43782f5738c59f5ac3c6fabcd6fcef
I20260927 21:39:41.562770 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f: Bootstrap starting.
I20260927 21:39:41.566200 6062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:41.567297 6062 log.cc:824] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:41.570173 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f: No bootstrap required, opened a new log
I20260927 21:39:41.573544 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:41.561080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.0.202.190:39457)
0927 21:39:41.562081 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:41.562093 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:41.562137 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:39:41.563893 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:41.563918 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:41.564520 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:41.564963 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:41.565542 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:41.566944 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:41.568346 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:41.569003 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:41.569242 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:41.570494 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:41.570504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:41.570532 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:41.571287 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:41.571388 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:41.571414 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:41.571560 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:41.571800 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 21:39:41.572146 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":351,"threads_started":1}
I20260927 21:39:41.577840 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:39:41.579592 6062 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39457 } }
I20260927 21:39:41.580001 6062 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:41.580108 6062 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e8f0d2a8c440b49d58321ae915f26f, State: Initialized, Role: FOLLOWER
I20260927 21:39:41.580730 6062 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39457 } }
I20260927 21:39:41.580971 6062 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:41.581115 6062 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:41.581290 6062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:41.583554 6062 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39457 } }
I20260927 21:39:41.584066 6062 leader_election.cc:302] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e8f0d2a8c440b49d58321ae915f26f; no voters:
I20260927 21:39:41.584700 6062 leader_election.cc:288] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:41.584837 6067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:41.585538 6067 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [term 1 LEADER]: Becoming Leader. State: Replica: e5e8f0d2a8c440b49d58321ae915f26f, State: Running, Role: LEADER
I20260927 21:39:41.586079 6067 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39457 } }
I20260927 21:39:41.587059 6062 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:41.589367 6068 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39457 } } }
I20260927 21:39:41.589344 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e8f0d2a8c440b49d58321ae915f26f. Latest consensus state: current_term: 1 leader_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e8f0d2a8c440b49d58321ae915f26f" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39457 } } }
I20260927 21:39:41.589754 6068 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:41.589815 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:41.596205 6074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:41.601749 6074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:41.610756 6074 catalog_manager.cc:1347] Generated new cluster ID: c526d63481744a47af4ca37fe2b4fec9
I20260927 21:39:41.610879 6074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:41.633383 6074 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:41.635188 6074 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:41.643855 6074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5e8f0d2a8c440b49d58321ae915f26f: Generated new TSK 0
I20260927 21:39:41.644917 6074 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:39:41.678112 6092 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 21:39:41.678288 6092 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:39:41.679388 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:41.658810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.0.202.190:39457)
0927 21:39:41.658999 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:41.659006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:41.659424 (+ 418us) server_negotiation.cc:407] Connection header received
0927 21:39:41.659763 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:41.659768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:41.659836 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:41.659984 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:39:41.660591 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:41.661318 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:41.662726 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:41.663169 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:41.663298 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:41.663804 (+ 506us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:39:41.678712 (+ 14908us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:39:41.679098 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 21:39:41.679517 6065 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 21:39:41.680913 6091 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:41.658674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:39457 (local address 127.0.0.1:53134)
0927 21:39:41.659319 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:41.659335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:41.659529 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:41.660042 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:41.660048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:41.660061 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:39:41.660441 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:41.660451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:41.661469 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:41.661476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:41.662578 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:41.662588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:41.663363 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:41.663370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:41.663502 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:41.663558 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:39:41.678993 (+ 15435us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:39:41.679334 (+ 341us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.202.190:39457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":367,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.202.190:39457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 21:39:41.682585 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 5996
I20260927 21:39:41.683737 6057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:41.777151 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 5996
2026-09-27T21:39:41Z chronyd exiting
I20260927 21:39:41.828501 810 test_util.cc:175] -----------------------------------------------
I20260927 21:39:41.828678 810 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790545135802652-810-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (765 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 21:39:41.832620 810 mini_oidc.cc:152] Starting JWKS server
I20260927 21:39:41.834445 810 webserver.cc:460] Webserver started at https://127.0.0.1:36771/ using document root <none> and password file <none>
I20260927 21:39:41.834692 810 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:39:41.835353 810 webserver.cc:460] Webserver started at http://127.0.0.1:40611/ using document root <none> and password file <none>
2026-09-27T21:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:41Z Disabled control of system clock
I20260927 21:39:41.863197 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:45557
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:38609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:45557
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/testchainca.cert with env {}
W20260927 21:39:42.198510 6099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:42.198807 6099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:42.198896 6099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:42.206174 6099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:42.206305 6099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:42.206344 6099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:42.206393 6099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:42.206446 6099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:39:42.206504 6099 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:39:42.206544 6099 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36771/jwks.json
I20260927 21:39:42.216595 6099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:38609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:45557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:42.217837 6099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:42.219105 6099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:42.231316 6104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:42.231252 6105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:42.231292 6107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:42.232427 6099 server_base.cc:1034] running on GCE node
I20260927 21:39:42.233285 6099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:42.234737 6099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:42.235977 6099 hybrid_clock.cc:648] HybridClock initialized: now 1790545182235938 us; error 40 us; skew 500 ppm
I20260927 21:39:42.236394 6099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:42.238868 6099 webserver.cc:460] Webserver started at http://127.0.202.190:46023/ using document root <none> and password file <none>
I20260927 21:39:42.239640 6099 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:42.239776 6099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:42.240099 6099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:42.243137 6099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "6f43a2ccba7b415095aa19cba3ae173a"
format_stamp: "Formatted at 2026-09-27 21:39:42 on dist-test-slave-25dq"
server_key: "e1f7f1e1de81db8fdf91ce8db276fdf7"
server_key_iv: "df708e06fa4e8612784e148409c9ad92"
server_key_version: "encryptionkey@0"
I20260927 21:39:42.243985 6099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "6f43a2ccba7b415095aa19cba3ae173a"
format_stamp: "Formatted at 2026-09-27 21:39:42 on dist-test-slave-25dq"
server_key: "e1f7f1e1de81db8fdf91ce8db276fdf7"
server_key_iv: "df708e06fa4e8612784e148409c9ad92"
server_key_version: "encryptionkey@0"
I20260927 21:39:42.249464 6099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:39:42.253332 6113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:42.254947 6099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:42.255362 6099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "6f43a2ccba7b415095aa19cba3ae173a"
format_stamp: "Formatted at 2026-09-27 21:39:42 on dist-test-slave-25dq"
server_key: "e1f7f1e1de81db8fdf91ce8db276fdf7"
server_key_iv: "df708e06fa4e8612784e148409c9ad92"
server_key_version: "encryptionkey@0"
I20260927 21:39:42.255657 6099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:42.276500 6099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:42.277410 6099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:42.277732 6099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:42.297003 6099 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:45557
I20260927 21:39:42.297044 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:45557 every 8 connection(s)
I20260927 21:39:42.298391 6099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:42.302186 6165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:42.303720 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6099
I20260927 21:39:42.303922 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:42.304765 810 external_mini_cluster.cc:1428] Setting key cbdddbcbf4abf1a5f5bbe4a7985cd7dd
I20260927 21:39:42.316529 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a: Bootstrap starting.
I20260927 21:39:42.318624 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:42.306726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.202.190:45557)
0927 21:39:42.307825 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:42.307838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:42.307887 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:39:42.309540 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:42.309565 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:42.310065 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:42.310402 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:42.311043 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:42.312396 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:42.314196 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:42.314862 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:42.314992 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:42.316251 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:42.316258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:42.316272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:42.316916 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:42.317002 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:42.317026 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:42.317118 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:42.317331 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:39:42.317665 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":434,"threads_started":1}
I20260927 21:39:42.319873 6165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:42.321137 6165 log.cc:824] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:42.322826 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:39:42.324900 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a: No bootstrap required, opened a new log
I20260927 21:39:42.333158 6165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43a2ccba7b415095aa19cba3ae173a" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 45557 } }
I20260927 21:39:42.333627 6165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:42.333770 6165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f43a2ccba7b415095aa19cba3ae173a, State: Initialized, Role: FOLLOWER
I20260927 21:39:42.334389 6165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43a2ccba7b415095aa19cba3ae173a" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 45557 } }
I20260927 21:39:42.334649 6165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:42.334803 6165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:42.334996 6165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:42.337430 6165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43a2ccba7b415095aa19cba3ae173a" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 45557 } }
I20260927 21:39:42.337981 6165 leader_election.cc:302] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f43a2ccba7b415095aa19cba3ae173a; no voters:
I20260927 21:39:42.338608 6165 leader_election.cc:288] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:42.338797 6170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:42.339982 6170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [term 1 LEADER]: Becoming Leader. State: Replica: 6f43a2ccba7b415095aa19cba3ae173a, State: Running, Role: LEADER
I20260927 21:39:42.340559 6170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43a2ccba7b415095aa19cba3ae173a" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 45557 } }
I20260927 21:39:42.341013 6165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:42.343602 6171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f43a2ccba7b415095aa19cba3ae173a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43a2ccba7b415095aa19cba3ae173a" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 45557 } } }
I20260927 21:39:42.343634 6172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f43a2ccba7b415095aa19cba3ae173a. Latest consensus state: current_term: 1 leader_uuid: "6f43a2ccba7b415095aa19cba3ae173a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f43a2ccba7b415095aa19cba3ae173a" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 45557 } } }
I20260927 21:39:42.344080 6171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:42.344089 6172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:42.352082 6177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:42.358439 6177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:42.368368 6177 catalog_manager.cc:1347] Generated new cluster ID: db22cb389b314fbcbbdce56676e66bf1
I20260927 21:39:42.368525 6177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:42.403995 6177 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:42.406004 6177 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:42.413882 6177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f43a2ccba7b415095aa19cba3ae173a: Generated new TSK 0
I20260927 21:39:42.414971 6177 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:39:42.453716 6195 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 21:39:42.453856 6195 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:39:42.455132 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:42.434490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.0.202.190:45557)
0927 21:39:42.434691 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:42.434699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:42.435072 (+ 373us) server_negotiation.cc:407] Connection header received
0927 21:39:42.435302 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:42.435309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:42.435405 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:42.435575 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:39:42.436062 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:42.437271 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:42.438633 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:42.439342 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:42.439473 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:42.440012 (+ 539us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:39:42.454445 (+ 14433us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:39:42.454820 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 21:39:42.455185 6194 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:42.434334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:45557 (local address 127.0.0.1:33340)
0927 21:39:42.434949 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:42.434983 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:42.435111 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:42.435614 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:42.435618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:42.435628 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:39:42.435906 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:42.435914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:42.437421 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:42.437425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:42.438493 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:42.438505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:42.439524 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:42.439529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:42.439695 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:42.439733 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:39:42.454694 (+ 14961us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:39:42.454895 (+ 201us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.202.190:45557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":333,"threads_started":1}
W20260927 21:39:42.455260 6168 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 21:39:42.456545 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6099
I20260927 21:39:42.457795 6160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:42.553090 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6099
2026-09-27T21:39:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (778 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 21:39:42.612177 810 mini_oidc.cc:152] Starting JWKS server
I20260927 21:39:42.614176 810 webserver.cc:460] Webserver started at https://127.0.0.1:46831/ using document root <none> and password file <none>
I20260927 21:39:42.614432 810 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:39:42.615099 810 webserver.cc:460] Webserver started at http://127.0.0.1:43507/ using document root <none> and password file <none>
2026-09-27T21:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:42Z Disabled control of system clock
I20260927 21:39:42.645038 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:36557
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:36557
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46831/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:39:42.977383 6202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:42.977739 6202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:42.977842 6202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:42.986306 6202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:42.986452 6202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:42.986500 6202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:42.986568 6202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:42.986624 6202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:39:42.986697 6202 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:39:42.986748 6202 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46831/jwks.json
I20260927 21:39:42.997429 6202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:36557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:36557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46831/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:42.998790 6202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:43.000787 6202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:43.013717 6207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:43.013742 6210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:43.013873 6208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:43.015007 6202 server_base.cc:1034] running on GCE node
I20260927 21:39:43.015870 6202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:43.017529 6202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:43.018764 6202 hybrid_clock.cc:648] HybridClock initialized: now 1790545183018694 us; error 57 us; skew 500 ppm
I20260927 21:39:43.019171 6202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:43.021879 6202 webserver.cc:460] Webserver started at http://127.0.202.190:37925/ using document root <none> and password file <none>
I20260927 21:39:43.022663 6202 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:43.022791 6202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:43.023191 6202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:43.026463 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "89c96d4998df4aafa0a878e771f9bbc2"
format_stamp: "Formatted at 2026-09-27 21:39:43 on dist-test-slave-25dq"
server_key: "35b452efaf4bb7b8cc8a08d4606bfdfb"
server_key_iv: "4c1d206df8eeb2dd56524af4f5478e3e"
server_key_version: "encryptionkey@0"
I20260927 21:39:43.027369 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "89c96d4998df4aafa0a878e771f9bbc2"
format_stamp: "Formatted at 2026-09-27 21:39:43 on dist-test-slave-25dq"
server_key: "35b452efaf4bb7b8cc8a08d4606bfdfb"
server_key_iv: "4c1d206df8eeb2dd56524af4f5478e3e"
server_key_version: "encryptionkey@0"
I20260927 21:39:43.033073 6202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:39:43.036959 6216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:43.038616 6202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:39:43.038851 6202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "89c96d4998df4aafa0a878e771f9bbc2"
format_stamp: "Formatted at 2026-09-27 21:39:43 on dist-test-slave-25dq"
server_key: "35b452efaf4bb7b8cc8a08d4606bfdfb"
server_key_iv: "4c1d206df8eeb2dd56524af4f5478e3e"
server_key_version: "encryptionkey@0"
I20260927 21:39:43.039105 6202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:43.073853 6202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:43.074692 6202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:43.074956 6202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:43.093685 6202 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:36557
I20260927 21:39:43.093704 6267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:36557 every 8 connection(s)
I20260927 21:39:43.095047 6202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:43.096463 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6202
I20260927 21:39:43.096683 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:43.097503 810 external_mini_cluster.cc:1428] Setting key 1f9e78c585619d92e6a022fe4a41d7d1
I20260927 21:39:43.100401 6268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:43.111275 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:43.099587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.0.202.190:36557)
0927 21:39:43.101012 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:43.101023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:43.101068 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:39:43.102618 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:43.102640 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:43.103130 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:43.103561 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:43.104128 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.105555 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.106869 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.107532 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.107719 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:43.108939 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:43.108948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:43.108967 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:43.109569 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:43.109647 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:43.109668 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:43.109820 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:43.110036 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:39:43.110345 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":671,"threads_started":1}
I20260927 21:39:43.111618 6268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2: Bootstrap starting.
I20260927 21:39:43.115100 6268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:43.115253 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:39:43.116192 6268 log.cc:824] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:43.119454 6268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2: No bootstrap required, opened a new log
I20260927 21:39:43.126812 6268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c96d4998df4aafa0a878e771f9bbc2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36557 } }
I20260927 21:39:43.127321 6268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:43.127449 6268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c96d4998df4aafa0a878e771f9bbc2, State: Initialized, Role: FOLLOWER
I20260927 21:39:43.128083 6268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c96d4998df4aafa0a878e771f9bbc2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36557 } }
I20260927 21:39:43.128328 6268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:43.128454 6268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:43.128623 6268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:43.130721 6268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c96d4998df4aafa0a878e771f9bbc2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36557 } }
I20260927 21:39:43.131220 6268 leader_election.cc:302] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89c96d4998df4aafa0a878e771f9bbc2; no voters:
I20260927 21:39:43.131790 6268 leader_election.cc:288] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:43.132000 6273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:43.133535 6273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 89c96d4998df4aafa0a878e771f9bbc2, State: Running, Role: LEADER
I20260927 21:39:43.134091 6268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:43.134153 6273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c96d4998df4aafa0a878e771f9bbc2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36557 } }
I20260927 21:39:43.137689 6275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c96d4998df4aafa0a878e771f9bbc2. Latest consensus state: current_term: 1 leader_uuid: "89c96d4998df4aafa0a878e771f9bbc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c96d4998df4aafa0a878e771f9bbc2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36557 } } }
I20260927 21:39:43.138093 6275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:43.138674 6274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89c96d4998df4aafa0a878e771f9bbc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c96d4998df4aafa0a878e771f9bbc2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36557 } } }
I20260927 21:39:43.139039 6274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:43.141760 6283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:43.146605 6283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:43.155453 6283 catalog_manager.cc:1347] Generated new cluster ID: 829319868647442896cc80bc291e02ce
I20260927 21:39:43.155725 6283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:43.167661 6283 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:43.169442 6283 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:43.178018 6283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89c96d4998df4aafa0a878e771f9bbc2: Generated new TSK 0
I20260927 21:39:43.178944 6283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:39:43.198766 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:43.190193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.0.202.190:36557)
0927 21:39:43.190355 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:43.190360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:43.190862 (+ 502us) server_negotiation.cc:407] Connection header received
0927 21:39:43.191079 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:43.191085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:43.191183 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:43.191361 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:39:43.191830 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.192986 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.194343 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.195058 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.195185 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:43.195658 (+ 473us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:39:43.198077 (+ 2419us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46831/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:39:43.198431 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46831/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 21:39:43.198908 6271 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46831/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 21:39:43.200646 6297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:43.190070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:36557 (local address 127.0.0.1:50776)
0927 21:39:43.190754 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:43.190771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:43.190891 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:43.191365 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:43.191368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:43.191377 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:39:43.191691 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:43.191700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.193131 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.193135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:43.194208 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:43.194219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.195231 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.195235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:43.195352 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:43.195383 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:39:43.198317 (+ 2934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46831/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:39:43.200180 (+ 1863us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.202.190:36557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46831/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":1520,"thread_start_us":421,"threads_started":1}
I20260927 21:39:43.202188 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6202
I20260927 21:39:43.203419 6263 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 21:39:43.210772 6298 webserver.cc:533] Webserver: sslize failed:
E20260927 21:39:43.210909 6298 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:39:43.299479 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6202
2026-09-27T21:39:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T21:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:43Z Disabled control of system clock
I20260927 21:39:43.378032 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33565
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:38217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33565
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/testchainca.cert with env {}
W20260927 21:39:43.705111 6303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:43.705438 6303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:43.705533 6303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:43.712883 6303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:43.713009 6303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:43.713048 6303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:43.713102 6303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:43.713147 6303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:39:43.713212 6303 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:39:43.713253 6303 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 21:39:43.724591 6303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:38217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:43.725967 6303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:43.727497 6303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:43.740530 6308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:43.740578 6309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:43.740597 6311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:43.741585 6303 server_base.cc:1034] running on GCE node
I20260927 21:39:43.742449 6303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:43.743929 6303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:43.745146 6303 hybrid_clock.cc:648] HybridClock initialized: now 1790545183745100 us; error 37 us; skew 500 ppm
I20260927 21:39:43.745496 6303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:39:43.747920 6303 webserver.cc:460] Webserver started at http://127.0.202.190:37759/ using document root <none> and password file <none>
I20260927 21:39:43.748776 6303 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:43.748900 6303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:43.749286 6303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:43.752496 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "3fd633fe902d4c8894cfe04765fda7bb"
format_stamp: "Formatted at 2026-09-27 21:39:43 on dist-test-slave-25dq"
server_key: "8c4351ed03f171165dd3ade0e8c81ee6"
server_key_iv: "4d6740672e62397dc551cb2d71ee07ba"
server_key_version: "encryptionkey@0"
I20260927 21:39:43.753384 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "3fd633fe902d4c8894cfe04765fda7bb"
format_stamp: "Formatted at 2026-09-27 21:39:43 on dist-test-slave-25dq"
server_key: "8c4351ed03f171165dd3ade0e8c81ee6"
server_key_iv: "4d6740672e62397dc551cb2d71ee07ba"
server_key_version: "encryptionkey@0"
I20260927 21:39:43.758852 6303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:39:43.762768 6317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:43.764453 6303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:43.764698 6303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "3fd633fe902d4c8894cfe04765fda7bb"
format_stamp: "Formatted at 2026-09-27 21:39:43 on dist-test-slave-25dq"
server_key: "8c4351ed03f171165dd3ade0e8c81ee6"
server_key_iv: "4d6740672e62397dc551cb2d71ee07ba"
server_key_version: "encryptionkey@0"
I20260927 21:39:43.764962 6303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:43.795046 6303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:43.796015 6303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:43.796362 6303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:43.817121 6303 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:33565
I20260927 21:39:43.817183 6368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:33565 every 8 connection(s)
I20260927 21:39:43.818589 6303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:43.822816 6369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:43.828679 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6303
I20260927 21:39:43.828895 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:43.829746 810 external_mini_cluster.cc:1428] Setting key a6697bc729db5b3c77f987cac2e234cc
I20260927 21:39:43.835618 6369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb: Bootstrap starting.
I20260927 21:39:43.839747 6369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:43.841262 6369 log.cc:824] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:43.844902 6369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb: No bootstrap required, opened a new log
I20260927 21:39:43.846243 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:43.832318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.0.202.190:33565)
0927 21:39:43.833673 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:43.833710 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:43.833757 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:39:43.835556 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:43.835581 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:43.836159 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:43.836599 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:43.837382 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.838848 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.841008 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.841959 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.842228 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:43.843329 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:43.843339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:43.843359 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:39:43.844098 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:43.844205 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:39:43.844236 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:43.844393 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:43.844611 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 21:39:43.844954 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":508,"threads_started":1}
I20260927 21:39:43.850761 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:39:43.853816 6369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd633fe902d4c8894cfe04765fda7bb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33565 } }
I20260927 21:39:43.854225 6369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:43.854334 6369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd633fe902d4c8894cfe04765fda7bb, State: Initialized, Role: FOLLOWER
I20260927 21:39:43.855352 6369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd633fe902d4c8894cfe04765fda7bb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33565 } }
I20260927 21:39:43.855696 6374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:43.856009 6374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd633fe902d4c8894cfe04765fda7bb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33565 } }
I20260927 21:39:43.855677 6369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:43.856760 6369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:43.856731 6374 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd633fe902d4c8894cfe04765fda7bb; no voters:
I20260927 21:39:43.856916 6369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:43.857695 6374 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 21:39:43.860212 6369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd633fe902d4c8894cfe04765fda7bb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33565 } }
I20260927 21:39:43.860764 6369 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd633fe902d4c8894cfe04765fda7bb; no voters:
I20260927 21:39:43.860836 6375 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 21:39:43.861128 6369 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:43.861353 6374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:43.862602 6374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [term 1 LEADER]: Becoming Leader. State: Replica: 3fd633fe902d4c8894cfe04765fda7bb, State: Running, Role: LEADER
I20260927 21:39:43.863183 6374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd633fe902d4c8894cfe04765fda7bb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33565 } }
I20260927 21:39:43.864101 6369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:43.865760 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fd633fe902d4c8894cfe04765fda7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd633fe902d4c8894cfe04765fda7bb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33565 } } }
I20260927 21:39:43.866185 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:43.865722 6377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fd633fe902d4c8894cfe04765fda7bb. Latest consensus state: current_term: 1 leader_uuid: "3fd633fe902d4c8894cfe04765fda7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd633fe902d4c8894cfe04765fda7bb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33565 } } }
I20260927 21:39:43.867290 6377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:43.868508 6380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:43.875568 6380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:43.885143 6380 catalog_manager.cc:1347] Generated new cluster ID: 6defe1f7124849ecb0b8257636d5f901
I20260927 21:39:43.885280 6380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:43.929546 6380 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:43.930939 6380 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:43.942236 6380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fd633fe902d4c8894cfe04765fda7bb: Generated new TSK 0
I20260927 21:39:43.943267 6380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:39:44.194149 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:43.969387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.0.202.190:33565)
0927 21:39:43.969563 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:43.969569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:43.970346 (+ 777us) server_negotiation.cc:407] Connection header received
0927 21:39:43.970695 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:43.970702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:43.970804 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:43.971008 (+ 204us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:39:43.971602 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.972625 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.974673 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.975179 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.975296 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:43.975951 (+ 655us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:39:44.193387 (+217436us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:39:44.193891 (+ 504us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46158: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 21:39:44.194305 6372 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46158: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 21:39:44.194608 6399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:43.969286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33565 (local address 127.0.0.1:46158)
0927 21:39:43.970232 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:43.970252 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:43.970412 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:43.971022 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:43.971027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:43.971041 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:39:43.971339 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:43.971350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.972789 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.972796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:43.974493 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:43.974511 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:43.975364 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:43.975371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:43.975582 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:43.975630 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:39:44.193835 (+218205us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:39:44.194162 (+ 327us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.202.190:33565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":520,"threads_started":1}
I20260927 21:39:44.196058 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6303
I20260927 21:39:44.197328 6364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:44.308218 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6303
2026-09-27T21:39:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (993 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (417 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (424 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:45 dist-test-slave-25dq krb5kdc[6414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:45 dist-test-slave-25dq krb5kdc[6414](info): set up 2 sockets
Sep 27 21:39:45 dist-test-slave-25dq krb5kdc[6414](info): commencing operation
krb5kdc: starting...
W20260927 21:39:46.289108 810 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:46 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:39:46.525597 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:42821
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:46.862599 6427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:46.862924 6427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:46.863013 6427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:46.870388 6427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:46.870519 6427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:46.870568 6427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:46.870622 6427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:46.870667 6427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:46.880777 6427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:42821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:46.882052 6427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:46.883299 6427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:46.896013 6435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:46.896085 6432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:46.896070 6433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:46.897239 6427 server_base.cc:1034] running on GCE node
I20260927 21:39:46.898037 6427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:46.899588 6427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:46.900849 6427 hybrid_clock.cc:648] HybridClock initialized: now 1790545186900786 us; error 54 us; skew 500 ppm
I20260927 21:39:46.901232 6427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:46 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:46.904515 6427 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:39:46.906267 6427 webserver.cc:460] Webserver started at http://127.0.202.190:42947/ using document root <none> and password file <none>
I20260927 21:39:46.906922 6427 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:46.907017 6427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:46.907294 6427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:46.910251 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "00b1ec2a662448d69ce3ccc3f288aa52"
format_stamp: "Formatted at 2026-09-27 21:39:46 on dist-test-slave-25dq"
server_key: "f6e835aa2d911cd80c8840d39fa431a2"
server_key_iv: "7be7de45f3e2e4b680d8c0c6e9ca4612"
server_key_version: "encryptionkey@0"
I20260927 21:39:46.910953 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "00b1ec2a662448d69ce3ccc3f288aa52"
format_stamp: "Formatted at 2026-09-27 21:39:46 on dist-test-slave-25dq"
server_key: "f6e835aa2d911cd80c8840d39fa431a2"
server_key_iv: "7be7de45f3e2e4b680d8c0c6e9ca4612"
server_key_version: "encryptionkey@0"
I20260927 21:39:46.916468 6427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 21:39:46.920348 6442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:46.921805 6427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:46.921960 6427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "00b1ec2a662448d69ce3ccc3f288aa52"
format_stamp: "Formatted at 2026-09-27 21:39:46 on dist-test-slave-25dq"
server_key: "f6e835aa2d911cd80c8840d39fa431a2"
server_key_iv: "7be7de45f3e2e4b680d8c0c6e9ca4612"
server_key_version: "encryptionkey@0"
I20260927 21:39:46.922133 6427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:46.936471 6427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:46.941501 6427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:46.941826 6427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:46.961500 6427 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:42821
I20260927 21:39:46.961539 6493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:42821 every 8 connection(s)
I20260927 21:39:46.962922 6427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:46.967027 6494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:46.967733 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6427
I20260927 21:39:46.967959 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:46.968911 810 external_mini_cluster.cc:1428] Setting key dcc21f8007bb36f226a26af9b58e1b88
I20260927 21:39:46.983495 6494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52: Bootstrap starting.
Sep 27 21:39:46 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:46.987634 6494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:46.989253 6494 log.cc:824] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:46.992810 6494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52: No bootstrap required, opened a new log
I20260927 21:39:46.995702 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:46.971208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.202.190:42821)
0927 21:39:46.972022 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:46.972034 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:46.972077 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:39:46.974216 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:46.974242 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:46.974668 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:46.975036 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:46.977047 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:46.978468 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:46.980498 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:46.981241 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:46.981384 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:46.986400 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:46.986414 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:46.986441 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:46.986499 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:46.990824 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:46.991572 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:46.991581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:46.991591 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:46.991826 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:46.992231 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:46.992237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:46.992240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:46.993008 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:46.993363 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:46.993769 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 21:39:46.994235 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":393,"threads_started":1}
I20260927 21:39:47.002367 6494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 42821 } }
I20260927 21:39:47.002851 6494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:47.003019 6494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b1ec2a662448d69ce3ccc3f288aa52, State: Initialized, Role: FOLLOWER
I20260927 21:39:47.003883 6494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 42821 } }
I20260927 21:39:47.004184 6494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:47.004376 6494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:47.004573 6494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:47.007002 6494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 42821 } }
I20260927 21:39:47.007570 6494 leader_election.cc:302] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00b1ec2a662448d69ce3ccc3f288aa52; no voters:
I20260927 21:39:47.008258 6494 leader_election.cc:288] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:47.008476 6499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:47.009644 6499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [term 1 LEADER]: Becoming Leader. State: Replica: 00b1ec2a662448d69ce3ccc3f288aa52, State: Running, Role: LEADER
I20260927 21:39:47.010128 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 42821 } }
I20260927 21:39:47.010756 6494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:47.012974 6500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 42821 } } }
I20260927 21:39:47.013466 6500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:47.012987 6501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00b1ec2a662448d69ce3ccc3f288aa52. Latest consensus state: current_term: 1 leader_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b1ec2a662448d69ce3ccc3f288aa52" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 42821 } } }
I20260927 21:39:47.014284 6501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52 [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:47.014370 6505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:47.022498 6505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:47.031716 6505 catalog_manager.cc:1347] Generated new cluster ID: 8e2a5d8bbc6b47c28d927f7179a50827
I20260927 21:39:47.031852 6505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:47.074455 6505 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:47.076658 6505 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:47.086071 6505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00b1ec2a662448d69ce3ccc3f288aa52: Generated new TSK 0
I20260927 21:39:47.087256 6505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:39:47.157738 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:42821
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:47.495769 6520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:47.496142 6520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:47.496349 6520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:47.503863 6520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:47.504032 6520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:47.504225 6520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:47.514735 6520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:47.516134 6520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:47.517655 6520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:47.533766 6525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:47.534041 6526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:47.534072 6528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:47.534727 6520 server_base.cc:1034] running on GCE node
I20260927 21:39:47.535471 6520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:47.536578 6520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:47.537899 6520 hybrid_clock.cc:648] HybridClock initialized: now 1790545187537806 us; error 77 us; skew 500 ppm
I20260927 21:39:47.538282 6520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:47 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:47.542125 6520 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:39:47.544128 6520 webserver.cc:460] Webserver started at http://127.0.202.129:38181/ using document root <none> and password file <none>
I20260927 21:39:47.545096 6520 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:47.545260 6520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:47.545600 6520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:47.549089 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "03f53de42e334042933c42bfd6c64ba4"
format_stamp: "Formatted at 2026-09-27 21:39:47 on dist-test-slave-25dq"
server_key: "7f03321a5cf0d754d4b882f00631b634"
server_key_iv: "1d1d4a2bef8b84ec885ace682e9320a9"
server_key_version: "encryptionkey@0"
I20260927 21:39:47.549983 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "03f53de42e334042933c42bfd6c64ba4"
format_stamp: "Formatted at 2026-09-27 21:39:47 on dist-test-slave-25dq"
server_key: "7f03321a5cf0d754d4b882f00631b634"
server_key_iv: "1d1d4a2bef8b84ec885ace682e9320a9"
server_key_version: "encryptionkey@0"
I20260927 21:39:47.556020 6520 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 21:39:47.560464 6535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:47.562202 6520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:39:47.562433 6520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "03f53de42e334042933c42bfd6c64ba4"
format_stamp: "Formatted at 2026-09-27 21:39:47 on dist-test-slave-25dq"
server_key: "7f03321a5cf0d754d4b882f00631b634"
server_key_iv: "1d1d4a2bef8b84ec885ace682e9320a9"
server_key_version: "encryptionkey@0"
I20260927 21:39:47.562680 6520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:47.584046 6520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:47.589165 6520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:47.589437 6520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:47.590370 6520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:47.591729 6520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:47.591807 6520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:47.591907 6520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:47.591951 6520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:47.627744 6520 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:44759
I20260927 21:39:47.627902 6647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:44759 every 8 connection(s)
I20260927 21:39:47.629179 6520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:47.632068 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6520
I20260927 21:39:47.632346 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:47.633019 810 external_mini_cluster.cc:1428] Setting key 5529183076dafd7efe92a8da2c1b9c1e
Sep 27 21:39:47 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:47.653967 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:47.631925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:56291 (local address 127.0.202.190:42821)
0927 21:39:47.632447 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:47.632454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:47.633966 (+ 1512us) server_negotiation.cc:407] Connection header received
0927 21:39:47.636563 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:47.636569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:47.636664 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:47.636789 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:47.638357 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:47.639488 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:47.641405 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:47.642061 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:47.642167 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:47.646958 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:47.646964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:47.646968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:47.647005 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:47.649947 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:47.651070 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:47.651079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:47.651085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:47.651258 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:47.651817 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:47.651825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:47.651831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:47.652252 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:47.652442 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:47.653266 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 21:39:47.653526 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":221,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:39:47.654719 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:47.632534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:42821 (local address 127.0.202.129:56291)
0927 21:39:47.633775 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:47.633827 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:47.636202 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:47.637014 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:47.637023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:47.637562 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:47.638177 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:47.638194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:47.639653 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:47.639658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:47.641251 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:47.641265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:47.642344 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:47.642353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:47.642733 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:47.643571 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:47.643612 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:47.646692 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:47.650120 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:47.650133 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:47.650155 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:47.650892 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:47.651397 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:47.651401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:47.651404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:47.651600 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:47.652442 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:47.652448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:47.652842 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 21:39:47.653079 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 21:39:47.653496 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":592,"threads_started":1}
I20260927 21:39:47.656759 6648 heartbeater.cc:347] Connected to a master server at 127.0.202.190:42821
I20260927 21:39:47.657128 6648 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:47.657925 6648 heartbeater.cc:511] Master 127.0.202.190:42821 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:39:47.660737 6459 ts_manager.cc:194] Registered new tserver with Master: 03f53de42e334042933c42bfd6c64ba4 (127.0.202.129:44759)
I20260927 21:39:47.662878 6459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:56291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:39:47.689349 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:42821
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:48.029755 6654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:48.030040 6654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:48.030208 6654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:48.037506 6654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:48.037622 6654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:48.037855 6654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:48.047768 6654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:48.049413 6654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:48.050788 6654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:48.065053 6659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:48.065130 6662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:48.065110 6660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:48.065765 6654 server_base.cc:1034] running on GCE node
I20260927 21:39:48.066463 6654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:48.067632 6654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:48.068893 6654 hybrid_clock.cc:648] HybridClock initialized: now 1790545188068838 us; error 40 us; skew 500 ppm
I20260927 21:39:48.069242 6654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.072669 6654 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:39:48.074189 6654 webserver.cc:460] Webserver started at http://127.0.202.130:42291/ using document root <none> and password file <none>
I20260927 21:39:48.074914 6654 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:48.075058 6654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:48.075397 6654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:48.078480 6654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "3333f6cfc23e4559bb40218726708bf3"
format_stamp: "Formatted at 2026-09-27 21:39:48 on dist-test-slave-25dq"
server_key: "9eb470606c9312c04328622ce394adca"
server_key_iv: "5782d5977673eeca08b275d4ef6e5c3c"
server_key_version: "encryptionkey@0"
I20260927 21:39:48.079624 6654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "3333f6cfc23e4559bb40218726708bf3"
format_stamp: "Formatted at 2026-09-27 21:39:48 on dist-test-slave-25dq"
server_key: "9eb470606c9312c04328622ce394adca"
server_key_iv: "5782d5977673eeca08b275d4ef6e5c3c"
server_key_version: "encryptionkey@0"
I20260927 21:39:48.085376 6654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:39:48.089587 6669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:48.091361 6654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:39:48.091514 6654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "3333f6cfc23e4559bb40218726708bf3"
format_stamp: "Formatted at 2026-09-27 21:39:48 on dist-test-slave-25dq"
server_key: "9eb470606c9312c04328622ce394adca"
server_key_iv: "5782d5977673eeca08b275d4ef6e5c3c"
server_key_version: "encryptionkey@0"
I20260927 21:39:48.091898 6654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:48.136776 6654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:48.141355 6654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:48.141606 6654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:48.142524 6654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:48.143846 6654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:48.143918 6654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:48.143997 6654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:48.144039 6654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:48.179545 6654 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:45465
I20260927 21:39:48.179602 6781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:45465 every 8 connection(s)
I20260927 21:39:48.181017 6654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:48.181861 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6654
I20260927 21:39:48.182068 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:48.182600 810 external_mini_cluster.cc:1428] Setting key b49e5a4a46b938ea69024806c9be87e0
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:39:48.205729 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.184160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:47479 (local address 127.0.202.190:42821)
0927 21:39:48.184662 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.184669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.185890 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 21:39:48.188871 (+ 2981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.188879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.188966 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.189069 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.191284 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.192257 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.194656 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.195135 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.195300 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.199705 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:48.199711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:48.199714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:48.199748 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:48.202431 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.203225 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.203231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.203234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.203366 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.203813 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.203820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.203823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.204101 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:48.204262 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:48.205083 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 21:39:48.205328 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":247,"threads_started":1}
I20260927 21:39:48.206943 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.184877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:42821 (local address 127.0.202.130:47479)
0927 21:39:48.185705 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:48.185769 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:48.188518 (+ 2749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:48.189361 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:48.189377 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:48.190219 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:48.191097 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:48.191126 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.192462 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.192467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:48.194416 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:48.194434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.195384 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.195394 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:48.195732 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.196639 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:48.196673 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:48.199442 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:48.202611 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:48.202621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:48.202641 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:48.203095 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:48.203485 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:48.203490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:48.203493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:48.203701 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:48.204239 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:48.204249 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:48.204666 (+ 417us) client_negotiation.cc:769] Sending connection context
0927 21:39:48.205002 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 21:39:48.205602 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":313,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:39:48.209080 6782 heartbeater.cc:347] Connected to a master server at 127.0.202.190:42821
I20260927 21:39:48.209438 6782 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:48.210247 6782 heartbeater.cc:511] Master 127.0.202.190:42821 requested a full tablet report, sending...
I20260927 21:39:48.212080 6459 ts_manager.cc:194] Registered new tserver with Master: 3333f6cfc23e4559bb40218726708bf3 (127.0.202.130:45465)
I20260927 21:39:48.213397 6459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:47479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:39:48.236966 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:42821
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:48.578006 6788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:48.578299 6788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:48.578469 6788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:48.585987 6788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:48.586102 6788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:48.586272 6788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:48.596482 6788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:48.597999 6788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:48.599282 6788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:48.613742 6793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:48.613902 6794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:48.613780 6796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:48.614837 6788 server_base.cc:1034] running on GCE node
I20260927 21:39:48.615468 6788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:48.616506 6788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:48.617776 6788 hybrid_clock.cc:648] HybridClock initialized: now 1790545188617709 us; error 51 us; skew 500 ppm
I20260927 21:39:48.618134 6788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.621464 6788 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:39:48.623016 6788 webserver.cc:460] Webserver started at http://127.0.202.131:42941/ using document root <none> and password file <none>
I20260927 21:39:48.623734 6788 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:48.623893 6788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:48.624212 6788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:48.627302 6788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "0bf5dee94c224866a060bff7f9a4faca"
format_stamp: "Formatted at 2026-09-27 21:39:48 on dist-test-slave-25dq"
server_key: "8f57772b3556c3c8079fe0b0f58af3ee"
server_key_iv: "ab03ff13438e0f711c65ccf4d81c6ab9"
server_key_version: "encryptionkey@0"
I20260927 21:39:48.628350 6788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "0bf5dee94c224866a060bff7f9a4faca"
format_stamp: "Formatted at 2026-09-27 21:39:48 on dist-test-slave-25dq"
server_key: "8f57772b3556c3c8079fe0b0f58af3ee"
server_key_iv: "ab03ff13438e0f711c65ccf4d81c6ab9"
server_key_version: "encryptionkey@0"
I20260927 21:39:48.635277 6788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:39:48.640077 6803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:48.641832 6788 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:39:48.642062 6788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "0bf5dee94c224866a060bff7f9a4faca"
format_stamp: "Formatted at 2026-09-27 21:39:48 on dist-test-slave-25dq"
server_key: "8f57772b3556c3c8079fe0b0f58af3ee"
server_key_iv: "ab03ff13438e0f711c65ccf4d81c6ab9"
server_key_version: "encryptionkey@0"
I20260927 21:39:48.642311 6788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:48.666529 6648 heartbeater.cc:503] Master 127.0.202.190:42821 was elected leader, sending a full tablet report...
I20260927 21:39:48.679508 6788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:48.684438 6788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:48.684742 6788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:48.685748 6788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:48.687187 6788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:48.687304 6788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:48.687459 6788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:48.687529 6788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:48.721875 6788 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:43341
I20260927 21:39:48.721961 6915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:43341 every 8 connection(s)
I20260927 21:39:48.723188 6788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:48.729081 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6788
I20260927 21:39:48.729306 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:48.729843 810 external_mini_cluster.cc:1428] Setting key a57d5d011f7ce9e22db5ca9adfa0d9c4
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:39:48.743463 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.725787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:41139 (local address 127.0.202.190:42821)
0927 21:39:48.726222 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.726227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.727294 (+ 1067us) server_negotiation.cc:407] Connection header received
0927 21:39:48.729191 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.729197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.729291 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.729409 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.731163 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.732208 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.733751 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.734128 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.734266 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.737842 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:48.737847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:48.737850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:48.737882 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:48.740453 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.741161 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.741167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.741172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.741288 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.741712 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.741717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.741720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.741985 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:48.742150 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:48.742856 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 21:39:48.743068 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":230,"threads_started":1}
I20260927 21:39:48.744721 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.726397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:42821 (local address 127.0.202.131:41139)
0927 21:39:48.727071 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:48.727121 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:48.728903 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:48.729674 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:48.729683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:48.730261 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:48.731000 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:48.731019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.732377 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.732381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:48.733608 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:48.733621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.734352 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.734358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:48.734584 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.735292 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:48.735319 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:48.737565 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:48.740583 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:48.740592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:48.740617 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:48.741026 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:48.741403 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:48.741408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:48.741411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:48.741590 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:48.742113 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:48.742122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:48.742483 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 21:39:48.742756 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 21:39:48.743297 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":267,"threads_started":1}
I20260927 21:39:48.746246 6916 heartbeater.cc:347] Connected to a master server at 127.0.202.190:42821
I20260927 21:39:48.746591 6916 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:48.747493 6916 heartbeater.cc:511] Master 127.0.202.190:42821 requested a full tablet report, sending...
I20260927 21:39:48.749039 6458 ts_manager.cc:194] Registered new tserver with Master: 0bf5dee94c224866a060bff7f9a4faca (127.0.202.131:43341)
I20260927 21:39:48.749919 6458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:41139
I20260927 21:39:48.759142 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/test-admin.keytab.
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.790768 810 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 21:39:48.796267 6924 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:48.793585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.0.202.129:44759)
0927 21:39:48.794355 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.794363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.794392 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:39:48.794454 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.794460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.794694 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.794840 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.795755 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59934: BlockingRecv error: recv got EOF from 127.0.0.1:59934 (error 108)
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":431,"threads_started":1}
W20260927 21:39:48.796417 810 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.798223 810 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.799638 810 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:48.824671 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.801493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.0.202.129:44759)
0927 21:39:48.801740 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.801746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.801831 (+ 85us) server_negotiation.cc:407] Connection header received
0927 21:39:48.801978 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.801982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.802045 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.802133 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.803095 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.804259 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.805647 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.806273 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.806378 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.817963 (+ 11585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:48.817978 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:48.817993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:48.818041 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:48.821493 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.822362 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.822369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.822371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.822495 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.822935 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.822939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.822941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.823384 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:48.823676 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:48.823964 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:39:48.824369 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 21:39:48.830843 6924 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:48.828950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.0.202.129:44759)
0927 21:39:48.829113 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.829117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.829497 (+ 380us) server_negotiation.cc:407] Connection header received
0927 21:39:48.829614 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.829617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.829738 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.829827 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.830560 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59946: BlockingRecv error: recv got EOF from 127.0.0.1:59946 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 21:39:48.831118 810 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.833035 810 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.834523 810 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:48.849542 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.836057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.0.202.129:44759)
0927 21:39:48.836273 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.836286 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.836472 (+ 186us) server_negotiation.cc:407] Connection header received
0927 21:39:48.836606 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.836609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.836666 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.836746 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.837664 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.838373 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.839865 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.840382 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.840483 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.844394 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:48.844400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:48.844404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:48.844438 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:48.847009 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.847724 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.847728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.847730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.847852 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.848174 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.848177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.848179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.848504 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:48.848701 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:48.849053 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 21:39:48.849238 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 21:39:48.854501 6924 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:39:48.852525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.0.202.129:44759)
0927 21:39:48.852837 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.852844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.853148 (+ 304us) server_negotiation.cc:407] Connection header received
0927 21:39:48.853336 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.853341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.853422 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.853525 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.854263 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59966: BlockingRecv error: recv got EOF from 127.0.0.1:59966 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
W20260927 21:39:48.854923 810 init.cc:244] Error while opening credential cache '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790545135802652-810-0/krb5kdc/krb5cc)
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.856599 810 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:48.858150 810 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:39:48.875109 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.859610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.0.202.129:44759)
0927 21:39:48.859991 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.859998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.860566 (+ 568us) server_negotiation.cc:407] Connection header received
0927 21:39:48.860678 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.860682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.860829 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.860938 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.862116 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.862907 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.864453 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.865005 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.865118 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.869325 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:48.869331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:48.869335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:48.869374 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:48.872117 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.872942 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.872947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.872949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.873120 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.873541 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.873546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.873548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.873913 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:48.874179 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:48.874477 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 21:39:48.874794 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260927 21:39:48.877283 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6520
I20260927 21:39:48.891165 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.878139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.0.202.129:44759)
0927 21:39:48.878360 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.878369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.878685 (+ 316us) server_negotiation.cc:407] Connection header received
0927 21:39:48.878803 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.878808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.878869 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.878951 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.879995 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.880877 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.882272 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.882872 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.882981 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:48.885360 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:48.885365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:48.885369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:48.885402 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:48.888460 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.889108 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.889115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.889122 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.889280 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:48.889722 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:48.889726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:48.889729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:48.890025 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:48.890306 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:48.890571 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 21:39:48.890739 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:39:48.891896 6643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:48.976737 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6520
I20260927 21:39:48.993388 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6654
Sep 27 21:39:48 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:39:49.010418 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:48.994413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.0.202.130:45465)
0927 21:39:48.994979 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:48.994988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:48.995016 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:39:48.995090 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:48.995097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:48.995329 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:48.995524 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:48.996782 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:48.997960 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:48.999406 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:49.000028 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:49.000209 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:49.004373 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:49.004386 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:49.004410 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:49.004453 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:49.007568 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:49.008355 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:49.008363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:49.008365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:49.008484 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:49.008854 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:49.008860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:49.008863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:49.009199 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:49.009503 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:49.009834 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 21:39:49.010100 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":296,"threads_started":1}
I20260927 21:39:49.011992 6777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:49.097551 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6654
I20260927 21:39:49.113867 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6788
Sep 27 21:39:49 dist-test-slave-25dq krb5kdc[6414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:39:49.131206 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:49.115133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.202.131:43341)
0927 21:39:49.115870 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:49.115877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:49.115906 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:39:49.115972 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:49.115979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:49.116191 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:49.116416 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:49.117878 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:49.119031 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:49.120523 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:49.121133 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:49.121290 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:49.125493 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:49.125506 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:49.125522 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:49.125563 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:49.128469 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:49.129219 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:49.129227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:49.129229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:49.129349 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:49.129731 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:49.129742 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:49.129745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:49.130068 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:49.130416 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:49.130642 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 21:39:49.130896 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":388,"threads_started":1}
I20260927 21:39:49.132541 6911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:49.215713 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6788
I20260927 21:39:49.233665 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6427
I20260927 21:39:49.234901 6489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:49.339404 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6427
2026-09-27T21:39:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4192 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:49 dist-test-slave-25dq krb5kdc[6944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:49 dist-test-slave-25dq krb5kdc[6944](info): set up 2 sockets
Sep 27 21:39:49 dist-test-slave-25dq krb5kdc[6944](info): commencing operation
krb5kdc: starting...
W20260927 21:39:50.481431 810 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:39:50 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:39:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab.
Entry for principal oryx/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab.
I20260927 21:39:50.712543 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:32975
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab
--principal=oryx/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:51.059042 6957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:51.059402 6957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:51.059580 6957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:51.067049 6957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:51.067175 6957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:51.067253 6957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:51.067338 6957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:51.067412 6957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:51.078891 6957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:32975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab
--principal=oryx/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:51.080406 6957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:51.081835 6957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:51.095180 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:51.095257 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:51.095212 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:51.096100 6957 server_base.cc:1034] running on GCE node
I20260927 21:39:51.096930 6957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:51.098371 6957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:51.099614 6957 hybrid_clock.cc:648] HybridClock initialized: now 1790545191099570 us; error 40 us; skew 500 ppm
I20260927 21:39:51.100088 6957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:51 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:51.103454 6957 init.cc:377] Logged in from keytab as oryx/127.0.202.190@KRBTEST.COM (short username oryx)
I20260927 21:39:51.105175 6957 webserver.cc:460] Webserver started at http://127.0.202.190:33347/ using document root <none> and password file <none>
I20260927 21:39:51.105877 6957 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:51.106020 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:51.106312 6957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:51.109359 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "f6fe6645ce17499397d98db51fc5732d"
format_stamp: "Formatted at 2026-09-27 21:39:51 on dist-test-slave-25dq"
server_key: "d389426cc6ffec631547fef1af897301"
server_key_iv: "611daae28c33ee06d8af497731b3a82e"
server_key_version: "encryptionkey@0"
I20260927 21:39:51.110123 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "f6fe6645ce17499397d98db51fc5732d"
format_stamp: "Formatted at 2026-09-27 21:39:51 on dist-test-slave-25dq"
server_key: "d389426cc6ffec631547fef1af897301"
server_key_iv: "611daae28c33ee06d8af497731b3a82e"
server_key_version: "encryptionkey@0"
I20260927 21:39:51.115694 6957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:39:51.119392 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:51.121054 6957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:39:51.121258 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "f6fe6645ce17499397d98db51fc5732d"
format_stamp: "Formatted at 2026-09-27 21:39:51 on dist-test-slave-25dq"
server_key: "d389426cc6ffec631547fef1af897301"
server_key_iv: "611daae28c33ee06d8af497731b3a82e"
server_key_version: "encryptionkey@0"
I20260927 21:39:51.121460 6957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:51.138526 6957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:51.142933 6957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:51.143282 6957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:51.162614 6957 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:32975
I20260927 21:39:51.162650 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:32975 every 8 connection(s)
I20260927 21:39:51.164064 6957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:39:51.167929 7024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:51.174280 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 6957
I20260927 21:39:51.174490 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:39:51.175293 810 external_mini_cluster.cc:1428] Setting key f9a36846ecd5c6493f6dd4db85a3592b
I20260927 21:39:51.179646 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: Bootstrap starting.
I20260927 21:39:51.182780 7024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:51.183837 7024 log.cc:824] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:51.187170 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: No bootstrap required, opened a new log
Sep 27 21:39:51 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:51.198129 7024 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } }
I20260927 21:39:51.198689 7024 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:51.198817 7024 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fe6645ce17499397d98db51fc5732d, State: Initialized, Role: FOLLOWER
I20260927 21:39:51.199594 7024 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } }
I20260927 21:39:51.199894 7024 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:39:51.200093 7024 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:39:51.200331 7024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:51.201960 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:51.177603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.0.202.190:32975)
0927 21:39:51.178483 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:51.178496 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:51.178540 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:39:51.180678 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:51.180706 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:51.181138 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:51.181579 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:51.183172 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:51.185045 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:51.186416 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:51.187570 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:51.187802 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:51.192186 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:51.192199 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:51.192226 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:51.192286 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:51.196784 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:51.197497 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:51.197506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:51.197516 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:51.197752 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:51.198228 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:51.198234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:51.198237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:51.199021 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:51.199482 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:51.199805 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 21:39:51.200364 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":374,"threads_started":1}
I20260927 21:39:51.203972 7024 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } }
I20260927 21:39:51.204593 7024 leader_election.cc:302] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6fe6645ce17499397d98db51fc5732d; no voters:
I20260927 21:39:51.205317 7024 leader_election.cc:288] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:39:51.205430 7029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:51.206248 7029 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 1 LEADER]: Becoming Leader. State: Replica: f6fe6645ce17499397d98db51fc5732d, State: Running, Role: LEADER
I20260927 21:39:51.206851 7029 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } }
I20260927 21:39:51.207712 7024 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:51.209611 7031 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6fe6645ce17499397d98db51fc5732d. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } }
I20260927 21:39:51.210063 7031 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:51.209969 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } }
I20260927 21:39:51.210579 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:51.211071 7037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:51.219234 7037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:51.228873 7037 catalog_manager.cc:1347] Generated new cluster ID: 72bf49c0e83246d8b17dcf907e65b68e
I20260927 21:39:51.229003 7037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:51.249004 7037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:39:51.250952 7037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:51.258528 7037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: Generated new TSK 0
I20260927 21:39:51.259470 7037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab.
Entry for principal oryx/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab.
I20260927 21:39:51.325227 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:32975
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab
--principal=oryx/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:51.663082 7050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:51.663370 7050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:51.663463 7050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:51.670750 7050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:51.670871 7050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:51.671015 7050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:51.680769 7050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab
--principal=oryx/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:51.682103 7050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:51.683362 7050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:51.698041 7056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:51.698155 7055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:51.698062 7058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:51.698834 7050 server_base.cc:1034] running on GCE node
I20260927 21:39:51.699501 7050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:51.700589 7050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:51.701840 7050 hybrid_clock.cc:648] HybridClock initialized: now 1790545191701756 us; error 66 us; skew 500 ppm
I20260927 21:39:51.702198 7050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:51 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:51.705745 7050 init.cc:377] Logged in from keytab as oryx/127.0.202.129@KRBTEST.COM (short username oryx)
I20260927 21:39:51.707247 7050 webserver.cc:460] Webserver started at http://127.0.202.129:39459/ using document root <none> and password file <none>
I20260927 21:39:51.708074 7050 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:51.708185 7050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:51.708519 7050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:51.711650 7050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "1d2fa69457ef4c37ab14b56b3f38b197"
format_stamp: "Formatted at 2026-09-27 21:39:51 on dist-test-slave-25dq"
server_key: "0d42574cee0110ab42fdd616b0c38ddf"
server_key_iv: "8d47e012a4a877c02344d6cac6bb1e0f"
server_key_version: "encryptionkey@0"
I20260927 21:39:51.712837 7050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "1d2fa69457ef4c37ab14b56b3f38b197"
format_stamp: "Formatted at 2026-09-27 21:39:51 on dist-test-slave-25dq"
server_key: "0d42574cee0110ab42fdd616b0c38ddf"
server_key_iv: "8d47e012a4a877c02344d6cac6bb1e0f"
server_key_version: "encryptionkey@0"
I20260927 21:39:51.718330 7050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:39:51.722286 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:51.723716 7050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:39:51.723910 7050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "1d2fa69457ef4c37ab14b56b3f38b197"
format_stamp: "Formatted at 2026-09-27 21:39:51 on dist-test-slave-25dq"
server_key: "0d42574cee0110ab42fdd616b0c38ddf"
server_key_iv: "8d47e012a4a877c02344d6cac6bb1e0f"
server_key_version: "encryptionkey@0"
I20260927 21:39:51.724130 7050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:51.742101 7050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:51.746356 7050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:51.746662 7050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:51.747648 7050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:51.749081 7050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:51.749153 7050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:51.749284 7050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:51.749357 7050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:51.783218 7050 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41049
I20260927 21:39:51.783263 7177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41049 every 8 connection(s)
I20260927 21:39:51.784569 7050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:51.787474 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 7050
I20260927 21:39:51.787681 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:39:51.788480 810 external_mini_cluster.cc:1428] Setting key 27687d66c42b3a8168d7fc3c9ae9a7f5
Sep 27 21:39:51 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:51.808007 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:51.787543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:46535 (local address 127.0.202.190:32975)
0927 21:39:51.788077 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:51.788083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:51.789305 (+ 1222us) server_negotiation.cc:407] Connection header received
0927 21:39:51.791737 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:51.791788 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:51.791864 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:51.791964 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:51.794189 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:51.795072 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:51.797623 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:51.798030 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:51.798176 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:51.801693 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:51.801704 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:51.801707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:51.801741 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:51.804654 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:51.805626 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:51.805637 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:51.805640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:51.805767 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:51.806216 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:51.806222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:51.806224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:51.806497 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:51.806650 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:51.807387 (+ 737us) server_negotiation.cc:299] Negotiation successful
0927 21:39:51.807609 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":300,"threads_started":1}
I20260927 21:39:51.809113 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:51.788243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.129:46535)
0927 21:39:51.789114 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:51.789168 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:51.791398 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:51.792275 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:51.792308 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:51.793125 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:51.793984 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:51.794012 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:51.795255 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:51.795263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:51.797462 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:51.797474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:51.798286 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:51.798293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:51.798532 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:51.799240 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:51.799264 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:51.801455 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:51.804894 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:51.804905 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:51.804924 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:51.805473 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:51.805882 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:51.805887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:51.805891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:51.806088 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:51.806626 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:51.806635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:51.806999 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 21:39:51.807272 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 21:39:51.807821 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for oryx/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:39:51.811038 7178 heartbeater.cc:347] Connected to a master server at 127.0.202.190:32975
I20260927 21:39:51.811442 7178 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:51.812529 7178 heartbeater.cc:511] Master 127.0.202.190:32975 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.130@KRBTEST.COM" created.
I20260927 21:39:51.815163 6989 ts_manager.cc:194] Registered new tserver with Master: 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049)
I20260927 21:39:51.816936 6989 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:46535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab.
Entry for principal oryx/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab.
I20260927 21:39:51.841383 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:32975
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab
--principal=oryx/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:52.177093 7184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:52.177371 7184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:52.177465 7184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:52.184926 7184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:52.185041 7184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:52.185154 7184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:52.195667 7184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab
--principal=oryx/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:52.196941 7184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:52.198199 7184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:52.211467 7192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:52.211449 7190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:52.212018 7189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:52.212420 7184 server_base.cc:1034] running on GCE node
I20260927 21:39:52.213132 7184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:52.214167 7184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:52.215396 7184 hybrid_clock.cc:648] HybridClock initialized: now 1790545192215348 us; error 35 us; skew 500 ppm
I20260927 21:39:52.215739 7184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:52 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:52.219074 7184 init.cc:377] Logged in from keytab as oryx/127.0.202.130@KRBTEST.COM (short username oryx)
I20260927 21:39:52.220636 7184 webserver.cc:460] Webserver started at http://127.0.202.130:34685/ using document root <none> and password file <none>
I20260927 21:39:52.221345 7184 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:52.221467 7184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:52.221769 7184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:52.224786 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "0827c759ab6f4b3fac139ee8d7c53da9"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "aadc7c4a891b51606103af9b1053c091"
server_key_iv: "ce6e42469055dc40ef86fd256a96290c"
server_key_version: "encryptionkey@0"
I20260927 21:39:52.225651 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "0827c759ab6f4b3fac139ee8d7c53da9"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "aadc7c4a891b51606103af9b1053c091"
server_key_iv: "ce6e42469055dc40ef86fd256a96290c"
server_key_version: "encryptionkey@0"
I20260927 21:39:52.231175 7184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:39:52.235008 7199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:52.236577 7184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:52.236809 7184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "0827c759ab6f4b3fac139ee8d7c53da9"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "aadc7c4a891b51606103af9b1053c091"
server_key_iv: "ce6e42469055dc40ef86fd256a96290c"
server_key_version: "encryptionkey@0"
I20260927 21:39:52.237032 7184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:52.251983 7184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:52.257314 7184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:52.257651 7184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:52.258674 7184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:52.260061 7184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:52.260170 7184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:52.260356 7184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:52.260435 7184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 21:39:52.294957 7184 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:45555
I20260927 21:39:52.295004 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:45555 every 8 connection(s)
I20260927 21:39:52.296242 7184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:52.297030 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 7184
I20260927 21:39:52.297232 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:39:52.297818 810 external_mini_cluster.cc:1428] Setting key 80f65660a3317b4a4b2985b13a79eabb
Sep 27 21:39:52 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:39:52.320686 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:52.299043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:38225 (local address 127.0.202.190:32975)
0927 21:39:52.299264 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:52.299270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:52.300606 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 21:39:52.303468 (+ 2862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:52.303473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:52.303549 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:52.303647 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:52.305869 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.306580 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.308393 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.309222 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.309381 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:52.313281 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:52.313291 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:52.313298 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:52.313346 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:52.317239 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:52.318116 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:52.318122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:52.318128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:52.318308 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:52.318722 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:52.318728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:52.318732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:52.319112 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:52.319282 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:52.319999 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 21:39:52.320243 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:39:52.321573 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:52.299622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:38225)
0927 21:39:52.300428 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:52.300494 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:52.303124 (+ 2630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:52.303935 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:52.303950 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:52.304779 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:52.305640 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:52.305668 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.306859 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.306865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:52.308159 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:52.308205 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.309537 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.309545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:52.309903 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:52.310634 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:52.310658 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:52.313005 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:52.317430 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:52.317439 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:52.317454 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:52.317962 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:52.318416 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:52.318419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:52.318421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:52.318600 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:52.319238 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:52.319245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:52.319596 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 21:39:52.319849 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 21:39:52.320247 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":269,"threads_started":1}
I20260927 21:39:52.323252 7312 heartbeater.cc:347] Connected to a master server at 127.0.202.190:32975
I20260927 21:39:52.323693 7312 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.131@KRBTEST.COM" created.
I20260927 21:39:52.324628 7312 heartbeater.cc:511] Master 127.0.202.190:32975 requested a full tablet report, sending...
I20260927 21:39:52.326354 6989 ts_manager.cc:194] Registered new tserver with Master: 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555)
I20260927 21:39:52.327581 6989 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:38225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab.
Entry for principal oryx/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab.
I20260927 21:39:52.353009 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:32975
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab
--principal=oryx/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:52.683066 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:52.683360 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:52.683455 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:52.691361 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:52.691556 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:52.691833 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:52.702960 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab
--principal=oryx/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:52.704449 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:52.705852 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:52.719854 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:52.719828 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:52.719825 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:52.720984 7317 server_base.cc:1034] running on GCE node
I20260927 21:39:52.721529 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:52.722554 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:52.723923 7317 hybrid_clock.cc:648] HybridClock initialized: now 1790545192723849 us; error 56 us; skew 500 ppm
I20260927 21:39:52.724231 7317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:52 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:52.727629 7317 init.cc:377] Logged in from keytab as oryx/127.0.202.131@KRBTEST.COM (short username oryx)
I20260927 21:39:52.729199 7317 webserver.cc:460] Webserver started at http://127.0.202.131:40995/ using document root <none> and password file <none>
I20260927 21:39:52.729902 7317 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:52.730046 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:52.730348 7317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:52.733282 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "6aa501a5d12a4771ae8ac0bb76970823"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "9cdefdd0f5015210c2a73d7bb923ec2d"
server_key_iv: "656c83e56ef161b80973146c8263dcd3"
server_key_version: "encryptionkey@0"
I20260927 21:39:52.734059 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "6aa501a5d12a4771ae8ac0bb76970823"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "9cdefdd0f5015210c2a73d7bb923ec2d"
server_key_iv: "656c83e56ef161b80973146c8263dcd3"
server_key_version: "encryptionkey@0"
I20260927 21:39:52.739471 7317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:39:52.743328 7332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:52.744935 7317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:52.745117 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "6aa501a5d12a4771ae8ac0bb76970823"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "9cdefdd0f5015210c2a73d7bb923ec2d"
server_key_iv: "656c83e56ef161b80973146c8263dcd3"
server_key_version: "encryptionkey@0"
I20260927 21:39:52.745321 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:52.756232 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:52.760988 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:52.761298 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:52.762259 7317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:52.763659 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:52.763774 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:52.763926 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:52.763999 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:52.798369 7317 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:33469
I20260927 21:39:52.798399 7444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:33469 every 8 connection(s)
I20260927 21:39:52.799594 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:52.804134 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 7317
I20260927 21:39:52.804356 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:39:52.804904 810 external_mini_cluster.cc:1428] Setting key b6f4d7fadf2b783ae88d17519309c607
Sep 27 21:39:52 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:52.820178 7178 heartbeater.cc:503] Master 127.0.202.190:32975 was elected leader, sending a full tablet report...
I20260927 21:39:52.822952 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:52.802639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:47375 (local address 127.0.202.190:32975)
0927 21:39:52.802868 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:52.802873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:52.804236 (+ 1363us) server_negotiation.cc:407] Connection header received
0927 21:39:52.806577 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:52.806584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:52.806709 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:52.806826 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:52.808968 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.809842 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.811654 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.812098 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.812196 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:52.816022 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:52.816031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:52.816034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:52.816072 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:52.819055 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:52.820058 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:52.820064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:52.820067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:52.820227 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:52.820954 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:52.820959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:52.820962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:52.821336 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:52.821477 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:52.822434 (+ 957us) server_negotiation.cc:299] Negotiation successful
0927 21:39:52.822614 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 21:39:52.823858 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:52.803173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.131:47375)
0927 21:39:52.804049 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:52.804097 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:52.806254 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:52.807162 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:52.807171 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:52.807959 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:52.808713 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:52.808731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.810012 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.810017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:52.811490 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:52.811502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:52.812358 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:52.812368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:52.812624 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:52.813433 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:52.813457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:52.815798 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:52.819227 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:52.819237 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:52.819256 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:52.819898 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:52.820444 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:52.820448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:52.820450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:52.820707 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:52.821491 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:52.821498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:52.821940 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 21:39:52.822288 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 21:39:52.822719 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":313,"threads_started":1}
I20260927 21:39:52.825489 7445 heartbeater.cc:347] Connected to a master server at 127.0.202.190:32975
I20260927 21:39:52.825909 7445 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:52.826730 7445 heartbeater.cc:511] Master 127.0.202.190:32975 requested a full tablet report, sending...
I20260927 21:39:52.828280 6988 ts_manager.cc:194] Registered new tserver with Master: 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:52.829351 6988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:47375
I20260927 21:39:52.833799 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.202.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.189.keytab.
Entry for principal oryx/127.0.202.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.189.keytab.
I20260927 21:39:52.888244 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:43383
--webserver_interface=127.0.202.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.189.keytab
--principal=oryx/127.0.202.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:32975,127.0.202.189:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:53.221721 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:53.222121 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:53.222288 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:53.229947 7450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:53.230100 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:53.230211 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:53.230319 7450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:53.230391 7450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:53.240659 7450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:32975,127.0.202.189:43383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.189.keytab
--principal=oryx/127.0.202.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.202.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
W20260927 21:39:53.241076 7450 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.202.190:32975,127.0.202.189:43383'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 21:39:53.242161 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:53.243616 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:53.256111 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:53.256145 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:53.256208 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:53.257207 7450 server_base.cc:1034] running on GCE node
I20260927 21:39:53.258106 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:53.260008 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:53.261345 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790545193261274 us; error 36 us; skew 500 ppm
I20260927 21:39:53.261825 7450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:53 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:53.265556 7450 init.cc:377] Logged in from keytab as oryx/127.0.202.189@KRBTEST.COM (short username oryx)
I20260927 21:39:53.267398 7450 webserver.cc:460] Webserver started at http://127.0.202.189:36329/ using document root <none> and password file <none>
I20260927 21:39:53.268285 7450 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:53.268469 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:53.268844 7450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:53.272285 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/instance:
uuid: "a8e0b28ddf5e489cbc201f40a46d8edd"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "cf90950ff22e06a75e55c393bf78cbdd"
server_key_iv: "92701093b139e17a8f38ea30d7c2d3ca"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.273422 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal/instance:
uuid: "a8e0b28ddf5e489cbc201f40a46d8edd"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "cf90950ff22e06a75e55c393bf78cbdd"
server_key_iv: "92701093b139e17a8f38ea30d7c2d3ca"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.282618 7450 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260927 21:39:53.287511 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:53.289181 7450 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 21:39:53.289412 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
uuid: "a8e0b28ddf5e489cbc201f40a46d8edd"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "cf90950ff22e06a75e55c393bf78cbdd"
server_key_iv: "92701093b139e17a8f38ea30d7c2d3ca"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.289649 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:53.315671 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:53.320668 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:53.320974 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:53.330675 7312 heartbeater.cc:503] Master 127.0.202.190:32975 was elected leader, sending a full tablet report...
I20260927 21:39:53.342900 7450 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.189:43383
I20260927 21:39:53.342908 7516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.189:43383 every 8 connection(s)
I20260927 21:39:53.344496 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
I20260927 21:39:53.348793 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:53.352141 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 7450
I20260927 21:39:53.357973 7517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:39:53 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.189@KRBTEST.COM
Sep 27 21:39:53 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.189@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:53.376932 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.360549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.0.202.190:32975)
0927 21:39:53.361037 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.361043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.361337 (+ 294us) server_negotiation.cc:407] Connection header received
0927 21:39:53.361522 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.361526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.361587 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.361702 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:53.363078 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.363963 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.365807 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.366186 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.366295 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.370305 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:53.370310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:53.370313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:53.370380 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:53.373526 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.374461 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.374470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.374473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.374596 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.375019 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.375023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.375025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.375455 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:53.375623 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:53.376455 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 21:39:53.376616 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
I20260927 21:39:53.378296 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.354666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.202.189:43383)
0927 21:39:53.355608 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.355620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.355664 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:39:53.357459 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.357486 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.357909 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.358332 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:53.359794 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.362047 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.363630 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.364606 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.364739 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.369665 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:53.369677 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:53.369695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:53.369735 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:53.372787 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.373707 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.373717 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.373726 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.374043 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.374435 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.374441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.374444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.375120 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:53.375326 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:53.376000 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 21:39:53.376533 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"spinlock_wait_cycles":27392,"thread_start_us":455,"threads_started":1}
I20260927 21:39:53.378316 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.360544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:34548)
0927 21:39:53.361208 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:53.361249 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:53.361377 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:53.361763 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:53.361769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:53.362614 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:53.362929 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.362940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.364105 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.364109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.365654 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.365668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.366425 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.366436 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.366675 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.367376 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:53.367406 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:53.370100 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:53.373707 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.373717 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.373729 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.374295 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.374703 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.374707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.374710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.374853 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.375689 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:53.375700 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:53.375996 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 21:39:53.376323 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 21:39:53.376642 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":1,"thread_start_us":334,"threads_started":1}
I20260927 21:39:53.381204 7517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:39:53 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.189@KRBTEST.COM for oryx/127.0.202.189@KRBTEST.COM
I20260927 21:39:53.395119 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.381974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.0.1:43960)
0927 21:39:53.382208 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:53.382232 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:53.382340 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:53.382647 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:53.382651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:53.383100 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:53.383233 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.383242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.384357 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.384362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.385324 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.385335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.386071 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.386077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.386315 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.386928 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:53.386947 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:53.389193 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:53.392416 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.392424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.392429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.392828 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.393234 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.393242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.393249 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.393443 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.394027 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:53.394032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:53.394187 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:39:53.394364 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 21:39:53.394687 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":3}
I20260927 21:39:53.395133 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.382142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.202.189:43383)
0927 21:39:53.382318 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.382324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.382342 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:39:53.382462 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.382466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.382530 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.382633 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:53.383370 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.384162 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.385467 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.385921 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.386063 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.389406 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:53.389411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:53.389415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:53.389444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:53.392245 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.392954 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.392962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.392966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.393091 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.393568 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.393573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.393577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.393909 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:53.394068 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:53.394439 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 21:39:53.394804 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":71}
I20260927 21:39:53.400775 7517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Bootstrap starting.
I20260927 21:39:53.404080 7517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:53.405287 7517 log.cc:824] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:53.408535 7517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: No bootstrap required, opened a new log
I20260927 21:39:53.415702 7517 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } }
I20260927 21:39:53.416069 7517 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:53.416167 7517 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e0b28ddf5e489cbc201f40a46d8edd, State: Initialized, Role: FOLLOWER
I20260927 21:39:53.416837 7517 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } }
I20260927 21:39:53.417685 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } } }
I20260927 21:39:53.418026 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:53.418720 7517 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:39:53.428089 7535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:39:53.428246 7535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:39:53.430155 7450 master_runner.cc:428] Detected that this master a8e0b28ddf5e489cbc201f40a46d8edd is joining an existing cluster
I20260927 21:39:53.430245 7450 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.202.189:43383
I20260927 21:39:53.458060 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.437696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.0.202.190:32975)
0927 21:39:53.437902 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.437910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.441001 (+ 3091us) server_negotiation.cc:407] Connection header received
0927 21:39:53.441290 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.441299 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.441428 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.441509 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:53.442794 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.443538 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.445940 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.446713 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.446822 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.450001 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:53.450010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:53.450018 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:53.450064 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:53.453915 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.454861 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.454866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.454870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.455295 (+ 425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.455838 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.455843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.455846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.456237 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:53.456434 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:53.457110 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 21:39:53.457661 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for oryx/127.0.202.188@KRBTEST.COM; defaulting to no policy
I20260927 21:39:53.459623 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.437715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:34552)
0927 21:39:53.440874 (+ 3159us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:53.440903 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:53.441133 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:53.441748 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:53.441753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:53.442311 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:53.442638 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.442647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.443695 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.443703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.445756 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.445773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.447164 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.447173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.447430 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.448143 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:53.448171 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:53.449775 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:53.454072 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.454080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.454087 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.454701 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.455476 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.455483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.455489 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.455688 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.456527 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:53.456533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:53.456785 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 21:39:53.458645 (+ 1860us) client_negotiation.cc:240] Negotiation successful
0927 21:39:53.459090 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2956,"thread_start_us":2624,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.188@KRBTEST.COM" created.
I20260927 21:39:53.463191 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.441082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.0.202.189:43383)
0927 21:39:53.441291 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.441296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.448217 (+ 6921us) server_negotiation.cc:407] Connection header received
0927 21:39:53.448656 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.448661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.448732 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.448807 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:53.450241 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.451377 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.453336 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.454026 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.454126 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.456516 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:53.456522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:53.456526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:53.456559 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:53.460234 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.461056 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.461061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.461065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.461210 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.461564 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.461570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.461578 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.461966 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:53.462110 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:53.462543 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 21:39:53.462788 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:39:53.464262 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.441400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.0.1:43972)
0927 21:39:53.448114 (+ 6714us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:53.448140 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:53.448526 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:53.449004 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:53.449008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:53.449771 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:53.450098 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.450109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.451584 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.451589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.453174 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.453188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.454424 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.454436 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.454647 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.455239 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:53.455263 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:53.456341 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:53.460400 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.460405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.460409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.460930 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.461320 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.461325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.461330 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.461460 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.462072 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:53.462077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:53.462220 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 21:39:53.463367 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0927 21:39:53.463695 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6542,"thread_start_us":6837,"threads_started":1}
Sep 27 21:39:53 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.190@KRBTEST.COM for oryx/127.0.202.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.188.keytab.
Entry for principal oryx/127.0.202.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.188.keytab.
I20260927 21:39:53.504863 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.467608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.0.202.189:43383)
0927 21:39:53.467790 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.467802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.486440 (+ 18638us) server_negotiation.cc:407] Connection header received
0927 21:39:53.487667 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.487674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.487804 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.487888 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:53.489732 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.490896 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.493012 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.493602 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.493781 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.497731 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:53.497740 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:53.497755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:53.497805 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:53.501493 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.502519 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.502525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.502529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.502698 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.503152 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.503157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.503160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.503526 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:53.503681 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:53.504156 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 21:39:53.504470 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:39:53.505002 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.467619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.0.1:43982)
0927 21:39:53.486238 (+ 18619us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:53.486296 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:53.487370 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:53.488220 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:53.488229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:53.489209 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:53.489535 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.489550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.491115 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.491122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.492842 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.492860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.493788 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.493794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.494005 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.494765 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:53.494783 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:53.497436 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:53.501692 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.501704 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.501711 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.502357 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.502838 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.502843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.502847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.503022 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.503668 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:53.503675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:53.503835 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 21:39:53.504007 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 21:39:53.504463 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18295,"thread_start_us":18856,"threads_started":1}
I20260927 21:39:53.505959 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:39799
--webserver_interface=127.0.202.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.188.keytab
--principal=oryx/127.0.202.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:39:53.509830 6988 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.202.189:43383: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.202.189"
port: 43383
}
attrs {
promote: true
}
}
dest_uuid: "f6fe6645ce17499397d98db51fc5732d"
cas_config_opid_index: -1
I20260927 21:39:53.511060 6988 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } }
I20260927 21:39:53.525247 7548 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a8e0b28ddf5e489cbc201f40a46d8edd (127.0.202.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.526705 7492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:53.530932 7492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Refusing update from remote peer f6fe6645ce17499397d98db51fc5732d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 21:39:53.541584 7549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:53.544521 7548 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.202.189:43383
I20260927 21:39:53.545308 7450 master.cc:537] Master@127.0.202.189:43383 shutting down...
I20260927 21:39:53.547183 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.547642 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:53.574704 7492 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a8e0b28ddf5e489cbc201f40a46d8edd (127.0.202.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.593083 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.593611 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: LEARNER
I20260927 21:39:53.593859 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.594080 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: LEARNER
I20260927 21:39:53.593657 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6fe6645ce17499397d98db51fc5732d. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.594203 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: LEARNER
I20260927 21:39:53.596704 7450 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 LEARNER]: Raft consensus shutting down.
I20260927 21:39:53.597218 7450 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 LEARNER]: Raft consensus is shut down!
I20260927 21:39:53.597388 7450 tablet_replica.cc:330] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: stopping tablet replica
I20260927 21:39:53.604980 7450 master.cc:552] Master@127.0.202.189:43383 shutdown complete.
I20260927 21:39:53.605087 7450 master_runner.cc:305] Clearing existing system tablet
I20260927 21:39:53.605525 7450 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:53.605620 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:53.608625 7450 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 21:39:53.610332 7561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:53.610698 7450 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:53.610956 7450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:39:53.611032 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
uuid: "a8e0b28ddf5e489cbc201f40a46d8edd"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "cf90950ff22e06a75e55c393bf78cbdd"
server_key_iv: "92701093b139e17a8f38ea30d7c2d3ca"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.615574 7450 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:53.622635 7450 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:39:53.622818 7450 log.cc:1196] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 21:39:53.623298 7450 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Deleting consensus metadata
I20260927 21:39:53.623584 7450 master_runner.cc:315] Copying system tablet from 127.0.202.190:32975
I20260927 21:39:53.633653 7450 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.202.190:32975
I20260927 21:39:53.649696 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.634533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:34554)
0927 21:39:53.635122 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:53.635145 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:53.635303 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:53.635711 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:53.635716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:53.636264 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:53.636572 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.636584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.638130 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.638136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.639710 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.639725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.640802 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.640809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.640976 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.641711 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:53.641751 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:53.642908 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:53.646696 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.646702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.646707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.647199 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.647646 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:53.647653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:53.647657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:53.647815 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:53.648504 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:53.648510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:53.648685 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 21:39:53.648855 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 21:39:53.649217 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":320,"threads_started":1}
I20260927 21:39:53.649727 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.634560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.0.202.190:32975)
0927 21:39:53.634747 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.634753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.635254 (+ 501us) server_negotiation.cc:407] Connection header received
0927 21:39:53.635452 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.635458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.635543 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.635651 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:53.636762 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.637946 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.639889 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.640629 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.640748 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:53.643124 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:53.643131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:53.643135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:53.643174 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:53.646548 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.647344 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.647349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.647353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.647521 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:53.647931 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:53.647936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:53.647939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:53.648268 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:53.648443 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:53.649031 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 21:39:53.649258 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:39:53.651157 7009 tablet_copy_service.cc:140] P f6fe6645ce17499397d98db51fc5732d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a8e0b28ddf5e489cbc201f40a46d8edd ({username='oryx', principal='oryx/127.0.202.189@KRBTEST.COM'} at 127.0.0.1:34554)
I20260927 21:39:53.651455 7009 tablet_copy_service.cc:161] P f6fe6645ce17499397d98db51fc5732d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a8e0b28ddf5e489cbc201f40a46d8edd at {username='oryx', principal='oryx/127.0.202.189@KRBTEST.COM'} at 127.0.0.1:34554: session id = a8e0b28ddf5e489cbc201f40a46d8edd-00000000000000000000000000000000
I20260927 21:39:53.656216 7009 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:39:53.658386 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:39:53.664947 7450 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: tablet copy: Starting download of 0 data blocks...
I20260927 21:39:53.665227 7450 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: tablet copy: Starting download of 1 WAL segments...
I20260927 21:39:53.667892 7450 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:39:53.672415 7009 tablet_copy_service.cc:342] P f6fe6645ce17499397d98db51fc5732d: Request end of tablet copy session a8e0b28ddf5e489cbc201f40a46d8edd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.202.189@KRBTEST.COM'} at 127.0.0.1:34554
I20260927 21:39:53.672642 7009 tablet_copy_service.cc:433] P f6fe6645ce17499397d98db51fc5732d: ending tablet copy session a8e0b28ddf5e489cbc201f40a46d8edd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a8e0b28ddf5e489cbc201f40a46d8edd
I20260927 21:39:53.684949 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:53.688151 7568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:53.689324 7569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:53.696372 7571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:53.696862 7450 server_base.cc:1034] running on GCE node
I20260927 21:39:53.697311 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:53.698017 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:53.699334 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790545193699267 us; error 32 us; skew 500 ppm
I20260927 21:39:53.699623 7450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:53 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:53.701958 7450 init.cc:377] Logged in from keytab as oryx/127.0.202.189@KRBTEST.COM (short username oryx)
I20260927 21:39:53.703305 7450 webserver.cc:460] Webserver started at http://127.0.202.189:38339/ using document root <none> and password file <none>
I20260927 21:39:53.703691 7450 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:53.703778 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:53.706853 7450 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 21:39:53.708959 7578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:53.709473 7450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:39:53.709669 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
uuid: "a8e0b28ddf5e489cbc201f40a46d8edd"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "cf90950ff22e06a75e55c393bf78cbdd"
server_key_iv: "92701093b139e17a8f38ea30d7c2d3ca"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.709923 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:53.733747 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:53.738284 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:53.774188 7450 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.189:43383
I20260927 21:39:53.774248 7629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.189:43383 every 8 connection(s)
I20260927 21:39:53.775265 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
I20260927 21:39:53.780628 7630 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:39:53.784699 7630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Bootstrap starting.
I20260927 21:39:53.804847 7630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:39:53.805438 7630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Bootstrap complete.
I20260927 21:39:53.806514 7630 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } }
I20260927 21:39:53.806777 7630 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a8e0b28ddf5e489cbc201f40a46d8edd, State: Initialized, Role: LEARNER
I20260927 21:39:53.807032 7630 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } }
I20260927 21:39:53.808068 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.808383 7630 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:53.808524 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: LEARNER
I20260927 21:39:53.820360 7644 catalog_manager.cc:1259] Loaded cluster ID: 72bf49c0e83246d8b17dcf907e65b68e
I20260927 21:39:53.820467 7644 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: loading cluster ID for follower catalog manager: success
I20260927 21:39:53.824591 7644 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: acquiring CA information for follower catalog manager: success
I20260927 21:39:53.827486 7644 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:39:53.832037 7445 heartbeater.cc:503] Master 127.0.202.190:32975 was elected leader, sending a full tablet report...
W20260927 21:39:53.929689 7546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:53.930003 7546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:53.930130 7546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:53.938088 7546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:39:53.938210 7546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:53.938306 7546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:53.938386 7546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:39:53.938450 7546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:39:53.948774 7546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.188.keytab
--principal=oryx/127.0.202.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:39799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.202.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:53.950312 7546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:53.951691 7546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:53.964409 7649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:53.964500 7650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:53.964522 7652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:53.965065 7546 server_base.cc:1034] running on GCE node
I20260927 21:39:53.965826 7546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:53.967235 7546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:53.968466 7546 hybrid_clock.cc:648] HybridClock initialized: now 1790545193968422 us; error 64 us; skew 500 ppm
I20260927 21:39:53.968832 7546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:53 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:53.977135 7546 init.cc:377] Logged in from keytab as oryx/127.0.202.188@KRBTEST.COM (short username oryx)
I20260927 21:39:53.978747 7546 webserver.cc:460] Webserver started at http://127.0.202.188:43457/ using document root <none> and password file <none>
I20260927 21:39:53.979456 7546 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:53.979560 7546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:53.979898 7546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:39:53.983108 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/instance:
uuid: "0bb4317543ea4b71a4b2217d432369aa"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "c2e9e670f1759b0135a59de9e0521444"
server_key_iv: "2fbcd86fadd2b02c21e89d4d17d656a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.983857 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal/instance:
uuid: "0bb4317543ea4b71a4b2217d432369aa"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "c2e9e670f1759b0135a59de9e0521444"
server_key_iv: "2fbcd86fadd2b02c21e89d4d17d656a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.989339 7546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:39:53.993237 7659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:53.993399 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:53.993673 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:53.995103 7546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:39:53.995306 7546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
uuid: "0bb4317543ea4b71a4b2217d432369aa"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "c2e9e670f1759b0135a59de9e0521444"
server_key_iv: "2fbcd86fadd2b02c21e89d4d17d656a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:53.995561 7546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:54.001689 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.993687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.0.1:43986)
0927 21:39:53.993968 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:53.993990 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:53.994112 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:53.995383 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:53.995388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:53.995417 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:53.995561 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.995591 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.997461 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.997466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:53.998939 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:53.998953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.001074 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.001080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.001225 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.001277 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.001367 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.001479 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 21:39:54.001935 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:53.993893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.0.202.189:43383)
0927 21:39:53.994985 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:53.994992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:53.995016 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:39:53.995091 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:53.995099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:53.995196 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:53.995329 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:53.995750 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:53.997277 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:53.999123 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.000835 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.000990 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.001251 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.001413 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.001552 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":730,"threads_started":1}
I20260927 21:39:54.003052 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6fe6645ce17499397d98db51fc5732d. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:54.003347 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: LEARNER
I20260927 21:39:54.003955 7548 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: attempting to promote NON_VOTER a8e0b28ddf5e489cbc201f40a46d8edd to VOTER
I20260927 21:39:54.004755 7548 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } }
I20260927 21:39:54.006327 7605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 LEARNER]: Refusing update from remote peer f6fe6645ce17499397d98db51fc5732d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:39:54.006861 7551 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 21:39:54.010023 7548 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a8e0b28ddf5e489cbc201f40a46d8edd (127.0.202.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } }
I20260927 21:39:54.012782 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } }
I20260927 21:39:54.012842 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } }
I20260927 21:39:54.013010 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.013101 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.013896 7605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a8e0b28ddf5e489cbc201f40a46d8edd (127.0.202.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } }
I20260927 21:39:54.017349 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } }
I20260927 21:39:54.017584 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:54.033975 7546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:54.038686 7546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:54.038983 7546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:54.058846 7546 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.188:39799
I20260927 21:39:54.058879 7712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.188:39799 every 8 connection(s)
I20260927 21:39:54.060235 7546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
I20260927 21:39:54.063710 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 7546
I20260927 21:39:54.064735 7713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:54.074122 7713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.188@KRBTEST.COM
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.188@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:54.088395 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.065955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.0.202.188:39799)
0927 21:39:54.066736 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.066748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.066789 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:39:54.069147 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.069170 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.069598 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.069953 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.071780 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.073171 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.074896 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.075572 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.075764 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.080671 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.080678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.080689 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.080723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.084243 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.084919 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.084928 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.084937 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.085142 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.085504 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.085511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.085517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.086042 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.086381 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.086819 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.087214 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":362,"threads_started":1}
I20260927 21:39:54.093231 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.076826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.0.202.190:32975)
0927 21:39:54.077104 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.077111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.078116 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 21:39:54.078323 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.078329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.078408 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.078521 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.079952 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.080935 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.082986 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.083416 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.083516 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.086971 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.086976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.086979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.087034 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.090570 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.091490 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.091494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.091497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.091620 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.091994 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.091997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.091999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.092249 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.092431 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.092844 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.093006 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 21:39:54.093261 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.076747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:34558)
0927 21:39:54.077531 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.077572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.078190 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.078584 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.078589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.079431 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.079750 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.079760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.081082 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.081087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.082827 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.082853 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.083602 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.083608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.083809 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.084444 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.084473 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.086757 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.090705 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.090713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.090718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.091391 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.091743 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.091747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.091750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.091897 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.092453 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.092458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.092599 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.092769 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.092995 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":386,"threads_started":1}
I20260927 21:39:54.094545 7713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.188@KRBTEST.COM for oryx/127.0.202.189@KRBTEST.COM
I20260927 21:39:54.107282 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.095162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.0.1:43996)
0927 21:39:54.095358 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.095372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.095445 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.095853 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.095856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.096152 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.096359 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.096376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.097334 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.097338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.098287 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.098298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.099023 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.099029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.099170 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.099833 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.099850 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.102080 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.105118 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.105121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.105124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.105510 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.105878 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.105882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.105885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.106037 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.106552 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.106556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.106695 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.106835 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.107057 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:39:54.107290 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.095256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.0.202.189:43383)
0927 21:39:54.095499 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.095506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.095527 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:39:54.095584 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.095589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.095662 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.095758 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.096500 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.097179 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.098438 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.098838 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.098944 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.102264 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.102269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.102273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.102304 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.104982 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.105610 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.105614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.105616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.105733 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.106156 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.106159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.106161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.106421 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.106525 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.106904 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.107062 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:39:54.108642 7713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.188@KRBTEST.COM for oryx/127.0.202.188@KRBTEST.COM
I20260927 21:39:54.121622 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.109534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.0.1:54052)
0927 21:39:54.110144 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.110162 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.110253 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.110564 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.110568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.110881 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.111023 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.111030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.112008 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.112011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.112885 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.112902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.113833 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.113838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.114106 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.114711 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.114729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.116738 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.119430 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.119434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.119437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.119780 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.120131 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.120134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.120137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.120274 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.120792 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.120795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.120974 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.121083 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.121267 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:39:54.121667 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.109534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.0.202.188:39799)
0927 21:39:54.110284 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.110289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.110303 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:39:54.110348 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.110351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.110436 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.110529 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.111131 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.111899 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.113114 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.113661 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.113819 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.116914 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.116919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.116922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.116951 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.119302 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.119897 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.119900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.119903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.120026 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.120409 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.120413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.120415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.120688 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.120810 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.121217 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.121399 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":670}
I20260927 21:39:54.126995 7713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Bootstrap starting.
I20260927 21:39:54.130571 7713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:54.132108 7713 log.cc:824] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:54.134917 7713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: No bootstrap required, opened a new log
I20260927 21:39:54.142860 7713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } }
I20260927 21:39:54.143213 7713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:54.143342 7713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb4317543ea4b71a4b2217d432369aa, State: Initialized, Role: FOLLOWER
I20260927 21:39:54.143970 7713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } }
I20260927 21:39:54.144932 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } } }
I20260927 21:39:54.145380 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:54.145987 7713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:39:54.155754 7730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:39:54.155908 7730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:39:54.161111 7546 master_runner.cc:428] Detected that this master 0bb4317543ea4b71a4b2217d432369aa is joining an existing cluster
I20260927 21:39:54.161265 7546 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.202.188:39799
I20260927 21:39:54.194999 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 7050
I20260927 21:39:54.200093 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.176377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.0.202.190:32975)
0927 21:39:54.176545 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.176549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.176566 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:39:54.176621 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.176625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.176691 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.176815 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.178304 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.179980 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.183034 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.184062 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.184192 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.188157 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.188169 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.188178 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.188235 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.196382 (+ 8147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.197285 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.197292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.197297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.197455 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.198010 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.198016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.198020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.198389 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.198523 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.199240 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.199503 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:39:54.202894 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.175042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:34566)
0927 21:39:54.175837 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.175861 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.176034 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.176852 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.176858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.177671 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.178087 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.178101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.180315 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.180323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.182803 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.182824 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.184577 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.184583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.184855 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.185939 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.185979 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.187839 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.196580 (+ 8741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.196585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.196590 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.197116 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.197621 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.197628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.197632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.197855 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.198649 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.198654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.198899 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.199618 (+ 719us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.202305 (+ 2687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":1892,"threads_started":1}
I20260927 21:39:54.203999 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.176344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.0.202.188:39799)
0927 21:39:54.176517 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.176523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.184901 (+ 8378us) server_negotiation.cc:407] Connection header received
0927 21:39:54.185142 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.185147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.185210 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.185285 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.186762 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.187942 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.189978 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.190716 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.190815 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.194819 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.194825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.194831 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.194863 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.200010 (+ 5147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.201126 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.201134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.201143 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.201315 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.201841 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.201848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.201854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.202369 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.202589 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.203206 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.203501 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:39:54.205384 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.175790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.0.1:54066)
0927 21:39:54.184797 (+ 9007us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.184818 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.184999 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.185469 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.185474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.186222 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.186603 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.186615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.188150 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.188155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.189821 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.189835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.191110 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.191116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.191286 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.193443 (+ 2157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.193468 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.194646 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.200178 (+ 5532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.200187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.200193 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.200957 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.201541 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.201548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.201554 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.201716 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.202678 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.202685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.202905 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.204307 (+ 1402us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.204715 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8841,"thread_start_us":8634,"threads_started":1}
I20260927 21:39:54.216090 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.177964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.0.202.189:43383)
0927 21:39:54.178161 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.178169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.191335 (+ 13166us) server_negotiation.cc:407] Connection header received
0927 21:39:54.191815 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.191823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.191913 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.191998 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.194086 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.195628 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.197851 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.198573 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.198674 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.207126 (+ 8452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.207136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.207143 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.207194 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.212432 (+ 5238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.213463 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.213470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.213474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.213680 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.214198 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.214203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.214207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.214633 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.214818 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.215459 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.215708 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:39:54.217156 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.177482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.0.1:44010)
0927 21:39:54.191209 (+ 13727us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.191234 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.191619 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.192250 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.192256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.193449 (+ 1193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.193884 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.193896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.195830 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.195835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.197672 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.197690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.198991 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.198998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.199250 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.200062 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.200091 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.202083 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.212596 (+ 10513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.212601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.212606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.213194 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.213848 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.213853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.213857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.214074 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.214899 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.214904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.215142 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.216326 (+ 1184us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.216649 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13541,"spinlock_wait_cycles":32128,"thread_start_us":13886,"threads_started":1}
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.190@KRBTEST.COM for oryx/127.0.202.188@KRBTEST.COM
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.129@KRBTEST.COM
I20260927 21:39:54.230733 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.210455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.0.1:54072)
0927 21:39:54.210815 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.210834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.210947 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.213149 (+ 2202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.213155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.213638 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.213895 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.213908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.215709 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.215717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.217234 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.217249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.218820 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.218828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.219001 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.219676 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.219702 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.222424 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.227465 (+ 5041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.227471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.227477 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.228001 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.228704 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.228710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.228714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.228857 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.229676 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.229682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.229885 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.230042 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.230341 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260927 21:39:54.231022 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.210685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.0.202.188:39799)
0927 21:39:54.212709 (+ 2024us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.212715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.212733 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:39:54.212834 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.212840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.212924 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.214051 (+ 1127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.214283 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.215524 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.217961 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.218617 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.221088 (+ 2471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.222627 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.222633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.222636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.222668 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.227287 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.228130 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.228137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.228141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.228545 (+ 404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.229110 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.229117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.229121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.229529 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.229686 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.230163 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.230424 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1895}
I20260927 21:39:54.232538 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.196072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.0.202.129:41049)
0927 21:39:54.209450 (+ 13378us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.209463 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.209500 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:39:54.209577 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.209588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.209931 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.210216 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.211944 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.213706 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.215863 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.217007 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.219231 (+ 2224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.223436 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.223461 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.223478 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.223513 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.228126 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.229078 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.229088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.229091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.229240 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.229735 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.229741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.229745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.230163 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.230487 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.231731 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.232036 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12585,"thread_start_us":12376,"threads_started":1}
I20260927 21:39:54.232594 6988 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.202.188:39799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa"
member_type: NON_VOTER
last_known_addr {
host: "127.0.202.188"
port: 39799
}
attrs {
promote: true
}
}
dest_uuid: "f6fe6645ce17499397d98db51fc5732d"
cas_config_opid_index: 6
I20260927 21:39:54.233320 6988 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } }
I20260927 21:39:54.235674 7173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:54.235668 7688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:54.239630 7688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 FOLLOWER]: Refusing update from remote peer f6fe6645ce17499397d98db51fc5732d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:39:54.240607 7548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:54.241671 7605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Refusing update from remote peer f6fe6645ce17499397d98db51fc5732d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:39:54.243021 7548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 21:39:54.245448 7549 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.246169 7605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.249616 7549 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.202.188:39799
I20260927 21:39:54.250269 7546 master.cc:537] Master@127.0.202.188:39799 shutting down...
I20260927 21:39:54.250248 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.250554 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:54.251259 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.251590 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.252271 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.252552 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.258844 7688 raft_consensus.cc:858] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } }
W20260927 21:39:54.259043 7688 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } }
I20260927 21:39:54.290616 7688 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a8e0b28ddf5e489cbc201f40a46d8edd (127.0.202.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:54.303225 7743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:54.303507 7743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 21:39:54.303856 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6fe6645ce17499397d98db51fc5732d. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:54.304081 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 21:39:54.304185 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: NON_VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: true } } }
I20260927 21:39:54.304548 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 21:39:54.305910 7546 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 21:39:54.305945 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.306217 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.306397 7546 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 21:39:54.306571 7546 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: stopping tablet replica
W20260927 21:39:54.307371 6973 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 21:39:54.314975 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.314504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.0.1:54080)
0927 21:39:54.314698 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.314812 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:39799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 21:39:54.315313 6973 consensus_peers.cc:487] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d -> Peer 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188:39799): Couldn't send request to peer 0bb4317543ea4b71a4b2217d432369aa. Status: Network error: Client connection negotiation failed: client connection to 127.0.202.188:39799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:39:54.325181 7546 master.cc:552] Master@127.0.202.188:39799 shutdown complete.
I20260927 21:39:54.325286 7546 master_runner.cc:305] Clearing existing system tablet
I20260927 21:39:54.325673 7546 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:54.325779 7546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:54.328833 7546 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 21:39:54.330722 7749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:54.331215 7546 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:54.331508 7546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 21:39:54.331616 7546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
uuid: "0bb4317543ea4b71a4b2217d432369aa"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "c2e9e670f1759b0135a59de9e0521444"
server_key_iv: "2fbcd86fadd2b02c21e89d4d17d656a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:54.337164 7546 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:54.345019 7546 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:39:54.345201 7546 log.cc:1196] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 21:39:54.345712 7546 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Deleting consensus metadata
I20260927 21:39:54.345983 7546 master_runner.cc:315] Copying system tablet from 127.0.202.190:32975
I20260927 21:39:54.357565 7546 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.202.190:32975
I20260927 21:39:54.374085 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.358531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.0.202.190:32975)
0927 21:39:54.358733 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.358740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.359290 (+ 550us) server_negotiation.cc:407] Connection header received
0927 21:39:54.359483 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.359489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.359585 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.359696 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:54.360799 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.361965 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.364249 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.364932 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.365069 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.367389 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:54.367396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:54.367400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:54.367460 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:54.370945 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.371805 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.371810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.371814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.371977 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:54.372450 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:54.372457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:54.372461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:54.372821 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:54.372977 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.373453 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.373727 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:39:54.374428 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.358446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:34572)
0927 21:39:54.359152 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.359172 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.359337 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.359755 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.359760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.360374 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:54.360635 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.360647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.362118 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.362123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.364086 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.364101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.365092 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.365098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.365293 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.366039 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:54.366060 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:54.367134 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:54.371116 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.371122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.371127 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.371661 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.372109 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:54.372116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:54.372121 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:54.372321 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:54.372950 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:54.372955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:54.373157 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.373334 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.373675 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":395,"threads_started":1}
I20260927 21:39:54.375092 7009 tablet_copy_service.cc:140] P f6fe6645ce17499397d98db51fc5732d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0bb4317543ea4b71a4b2217d432369aa ({username='oryx', principal='oryx/127.0.202.188@KRBTEST.COM'} at 127.0.0.1:34572)
I20260927 21:39:54.375260 7009 tablet_copy_service.cc:161] P f6fe6645ce17499397d98db51fc5732d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0bb4317543ea4b71a4b2217d432369aa at {username='oryx', principal='oryx/127.0.202.188@KRBTEST.COM'} at 127.0.0.1:34572: session id = 0bb4317543ea4b71a4b2217d432369aa-00000000000000000000000000000000
I20260927 21:39:54.378924 7009 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:39:54.381003 7546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:39:54.387746 7546 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: tablet copy: Starting download of 0 data blocks...
I20260927 21:39:54.388055 7546 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: tablet copy: Starting download of 1 WAL segments...
I20260927 21:39:54.390748 7546 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:39:54.393697 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 7050
I20260927 21:39:54.395888 7009 tablet_copy_service.cc:342] P f6fe6645ce17499397d98db51fc5732d: Request end of tablet copy session 0bb4317543ea4b71a4b2217d432369aa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.202.188@KRBTEST.COM'} at 127.0.0.1:34572
I20260927 21:39:54.396034 7009 tablet_copy_service.cc:433] P f6fe6645ce17499397d98db51fc5732d: ending tablet copy session 0bb4317543ea4b71a4b2217d432369aa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0bb4317543ea4b71a4b2217d432369aa
I20260927 21:39:54.407275 7546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:54.410226 7756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:54.416847 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:41049
--local_ip_for_outbound_sockets=127.0.202.129
--tserver_master_addrs=127.0.202.190:32975
--webserver_port=39459
--webserver_interface=127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab
--principal=oryx/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:54.419955 7757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:54.420635 7759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:54.421478 7546 server_base.cc:1034] running on GCE node
I20260927 21:39:54.421803 7546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:54.422484 7546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:54.423677 7546 hybrid_clock.cc:648] HybridClock initialized: now 1790545194423636 us; error 41 us; skew 500 ppm
I20260927 21:39:54.423974 7546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:54.426152 7546 init.cc:377] Logged in from keytab as oryx/127.0.202.188@KRBTEST.COM (short username oryx)
I20260927 21:39:54.433211 7546 webserver.cc:460] Webserver started at http://127.0.202.188:46653/ using document root <none> and password file <none>
I20260927 21:39:54.433524 7546 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:54.433641 7546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:54.436838 7546 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 21:39:54.439625 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:54.440125 7546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:39:54.440248 7546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
uuid: "0bb4317543ea4b71a4b2217d432369aa"
format_stamp: "Formatted at 2026-09-27 21:39:53 on dist-test-slave-25dq"
server_key: "c2e9e670f1759b0135a59de9e0521444"
server_key_iv: "2fbcd86fadd2b02c21e89d4d17d656a2"
server_key_version: "encryptionkey@0"
I20260927 21:39:54.440604 7546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:54.458186 7546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:54.462339 7546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:54.772028 7818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.188:39799 every 8 connection(s)
I20260927 21:39:54.772272 7546 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.188:39799
I20260927 21:39:54.773433 7546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
I20260927 21:39:54.796420 7819 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:39:54.800479 7819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Bootstrap starting.
I20260927 21:39:54.821246 7819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:39:54.821959 7819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: Bootstrap complete.
I20260927 21:39:54.823083 7819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } }
I20260927 21:39:54.823457 7819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0bb4317543ea4b71a4b2217d432369aa, State: Initialized, Role: LEARNER
I20260927 21:39:54.823758 7819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } }
I20260927 21:39:54.824618 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.825009 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: LEARNER
I20260927 21:39:54.825037 7819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:39:54.832944 7833 catalog_manager.cc:1259] Loaded cluster ID: 72bf49c0e83246d8b17dcf907e65b68e
I20260927 21:39:54.833115 7833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: loading cluster ID for follower catalog manager: success
I20260927 21:39:54.836196 7833 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: acquiring CA information for follower catalog manager: success
I20260927 21:39:54.837952 7833 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 21:39:54.913210 7762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:54.913575 7762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:54.913755 7762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:54.921452 7762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260927 21:39:54.921269 7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
W20260927 21:39:54.921638 7762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260927 21:39:54.921715 7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
W20260927 21:39:54.921902 7762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:39:54.934938 7762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab
--principal=oryx/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:41049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=39459
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:54.936509 7762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:54.938272 7762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:39:54.938406 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.921511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.0.1:54094)
0927 21:39:54.922018 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:54.922040 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:54.922176 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:54.932531 (+ 10355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:54.932536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:54.932569 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:54.932747 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.932756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.934616 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.934621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.936121 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:54.936138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.937729 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.937734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:54.937887 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.937915 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:39:54.938000 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:39:54.938107 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":443,"threads_started":1}
I20260927 21:39:54.938768 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:54.921792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.0.202.188:39799)
0927 21:39:54.932070 (+ 10278us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:54.932077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:54.932106 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:39:54.932187 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:54.932193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:54.932324 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:54.932476 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:54.932994 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.934437 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.936329 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:54.937531 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:54.937675 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:54.937908 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:54.938046 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:39:54.938181 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10135,"thread_start_us":10030,"threads_started":1}
I20260927 21:39:54.939713 7794 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260927 21:39:54.940188 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6fe6645ce17499397d98db51fc5732d. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: NON_VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: true } } }
I20260927 21:39:54.940526 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: LEARNER
I20260927 21:39:54.940928 7837 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: attempting to promote NON_VOTER 0bb4317543ea4b71a4b2217d432369aa to VOTER
I20260927 21:39:54.941776 7837 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } }
I20260927 21:39:54.943636 7794 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 LEARNER]: Refusing update from remote peer f6fe6645ce17499397d98db51fc5732d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 21:39:54.943872 7605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Refusing update from remote peer f6fe6645ce17499397d98db51fc5732d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 21:39:54.944157 7548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 21:39:54.944504 7837 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 21:39:54.946986 7548 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:54.949421 7605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:54.951138 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:54.951472 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.952212 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:54.952474 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.955889 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:54.956202 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:54.961053 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:54.961344 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:54.961922 7794 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
W20260927 21:39:54.965339 7844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:54.965373 7846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:54.966097 7848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:54.966185 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6fe6645ce17499397d98db51fc5732d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:54.966452 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:54.971843 7762 server_base.cc:1034] running on GCE node
I20260927 21:39:54.972719 7762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:54.973831 7762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:54.975075 7762 hybrid_clock.cc:648] HybridClock initialized: now 1790545194975028 us; error 33 us; skew 500 ppm
I20260927 21:39:54.975433 7762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:54 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:54.978770 7762 init.cc:377] Logged in from keytab as oryx/127.0.202.129@KRBTEST.COM (short username oryx)
I20260927 21:39:54.980621 7762 webserver.cc:460] Webserver started at http://127.0.202.129:39459/ using document root <none> and password file <none>
I20260927 21:39:54.981359 7762 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:54.981520 7762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:54.986943 7762 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 21:39:54.989797 7855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:54.991235 7762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:39:54.991420 7762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "1d2fa69457ef4c37ab14b56b3f38b197"
format_stamp: "Formatted at 2026-09-27 21:39:51 on dist-test-slave-25dq"
server_key: "0d42574cee0110ab42fdd616b0c38ddf"
server_key_iv: "8d47e012a4a877c02344d6cac6bb1e0f"
server_key_version: "encryptionkey@0"
I20260927 21:39:54.993114 7762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:55.016983 7762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:55.021476 7762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:55.021764 7762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:55.022745 7762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:55.024276 7762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:55.024456 7762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 21:39:55.024590 7762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:55.024690 7762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:55.063434 7762 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41049
I20260927 21:39:55.063469 7967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41049 every 8 connection(s)
I20260927 21:39:55.065269 7762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:39:55.071836 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 7762
I20260927 21:39:55.072206 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 7184
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.188@KRBTEST.COM
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.189@KRBTEST.COM
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:55.113116 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.068192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:54539 (local address 127.0.202.188:39799)
0927 21:39:55.068473 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.068482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.071669 (+ 3187us) server_negotiation.cc:407] Connection header received
0927 21:39:55.075350 (+ 3681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.075358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.075468 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.075556 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.088214 (+ 12658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.089964 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.092907 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.093918 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.094061 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.104423 (+ 10362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.104434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.104440 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.104488 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.108674 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.109839 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.109845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.109849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.110026 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.110661 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.110666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.110670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.111054 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.111191 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.112435 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.112708 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:39:55.113547 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.068081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:56519 (local address 127.0.202.189:43383)
0927 21:39:55.069664 (+ 1583us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.069669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.080163 (+ 10494us) server_negotiation.cc:407] Connection header received
0927 21:39:55.080547 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.080557 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.080664 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.080872 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.082624 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.084206 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.095753 (+ 11547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.097051 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.097205 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.105264 (+ 8059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.105271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.105274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.105310 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.109844 (+ 4534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.110869 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.110875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.110878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.111025 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.111570 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.111576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.111579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.111971 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.112112 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.112721 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.112973 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":1289,"threads_started":1}
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.130@KRBTEST.COM
I20260927 21:39:55.115844 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.069146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.202.129:54539)
0927 21:39:55.071431 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:55.071495 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:55.074860 (+ 3365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:55.076311 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:55.076331 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:55.077447 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:55.078641 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.078677 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.090243 (+ 11566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.090253 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.092668 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.092689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.094553 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.094564 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.095020 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.099111 (+ 4091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:55.099164 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:55.104104 (+ 4940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:55.108872 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.108882 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.108903 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.109670 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.110184 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.110190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.110194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.110505 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.111308 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:55.111322 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:55.111861 (+ 539us) client_negotiation.cc:769] Sending connection context
0927 21:39:55.113449 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0927 21:39:55.114114 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"thread_start_us":728,"threads_started":1}
I20260927 21:39:55.115877 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.069148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.202.129:56519)
0927 21:39:55.080010 (+ 10862us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:55.080039 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:55.080307 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:55.081216 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:55.081221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:55.081955 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:55.082375 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.082391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.084517 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.084526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.087883 (+ 3357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.087929 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.097616 (+ 9687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.097626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.098560 (+ 934us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.101573 (+ 3013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:55.101598 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:55.104996 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:55.110033 (+ 5037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.110038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.110041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.110669 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.111187 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.111192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.111197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.111433 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.112210 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:55.112219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:55.112435 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 21:39:55.113798 (+ 1363us) client_negotiation.cc:240] Negotiation successful
0927 21:39:55.114585 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10654,"mutex_wait_us":1026,"thread_start_us":9865,"threads_started":1}
I20260927 21:39:55.117779 7968 heartbeater.cc:347] Connected to a master server at 127.0.202.188:39799
I20260927 21:39:55.118147 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.086506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.129:51141)
0927 21:39:55.092875 (+ 6369us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:55.092904 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:55.093072 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:55.095280 (+ 2208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:55.095288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:55.096563 (+ 1275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:55.097077 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.097094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.101618 (+ 4524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.101625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.103398 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.103415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.104683 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.104689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.104883 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.105712 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:55.105734 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:55.108763 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:55.114109 (+ 5346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.114115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.114119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.115573 (+ 1454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.116064 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.116072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.116077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.116255 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.117029 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:55.117034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:55.117196 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 21:39:55.117328 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:39:55.117663 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6169,"thread_start_us":6643,"threads_started":1}
I20260927 21:39:55.118252 7970 heartbeater.cc:347] Connected to a master server at 127.0.202.189:43383
I20260927 21:39:55.118345 7970 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:55.118607 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.080417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:51141 (local address 127.0.202.190:32975)
0927 21:39:55.094536 (+ 14119us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.094544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.094572 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:39:55.094647 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.094655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.094755 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.094889 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.097408 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.100740 (+ 3332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.103597 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.104220 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.104385 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.109026 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.109034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.109039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.109083 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.113868 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.115744 (+ 1876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.115751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.115755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.115913 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.116459 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.116469 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.116475 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.116828 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.116957 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.117954 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.118187 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13927,"thread_start_us":14643,"threads_started":1}
I20260927 21:39:55.119151 7970 heartbeater.cc:511] Master 127.0.202.189:43383 requested a full tablet report, sending...
I20260927 21:39:55.118175 7968 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:55.120391 7969 heartbeater.cc:347] Connected to a master server at 127.0.202.190:32975
I20260927 21:39:55.120569 7969 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:55.120601 7968 heartbeater.cc:511] Master 127.0.202.188:39799 requested a full tablet report, sending...
I20260927 21:39:55.121034 7969 heartbeater.cc:511] Master 127.0.202.190:32975 requested a full tablet report, sending...
I20260927 21:39:55.122244 7595 ts_manager.cc:194] Registered new tserver with Master: 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049)
I20260927 21:39:55.122373 7784 ts_manager.cc:194] Registered new tserver with Master: 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049)
I20260927 21:39:55.123958 6988 ts_manager.cc:194] Re-registered known tserver with Master: 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049)
I20260927 21:39:55.125159 6988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:51141
I20260927 21:39:55.127028 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.073412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.0.202.130:45555)
0927 21:39:55.100547 (+ 27135us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.100557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.100589 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:39:55.100654 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.100663 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.101091 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.101280 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.103453 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.105090 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.107382 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.108558 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.108709 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.118184 (+ 9475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.118434 (+ 250us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.118455 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.118504 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.123068 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.124012 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.124020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.124023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.124215 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.124789 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.124794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.124796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.125488 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.125792 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.126254 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.126600 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26941,"thread_start_us":26737,"threads_started":1}
I20260927 21:39:55.128477 7307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:55.216563 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 7184
I20260927 21:39:55.232931 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:45555
--local_ip_for_outbound_sockets=127.0.202.130
--tserver_master_addrs=127.0.202.190:32975
--webserver_port=34685
--webserver_interface=127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab
--principal=oryx/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:39:55.574142 7980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:55.574433 7980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:55.574600 7980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:55.582031 7980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:55.582145 7980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:55.582324 7980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:39:55.592222 7980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab
--principal=oryx/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:45555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=34685
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:55.593659 7980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:55.595063 7980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:55.609755 7985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:55.609790 7988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:55.609864 7986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:55.610538 7980 server_base.cc:1034] running on GCE node
I20260927 21:39:55.611195 7980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:55.612283 7980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:55.613559 7980 hybrid_clock.cc:648] HybridClock initialized: now 1790545195613493 us; error 60 us; skew 500 ppm
I20260927 21:39:55.613929 7980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:55.617301 7980 init.cc:377] Logged in from keytab as oryx/127.0.202.130@KRBTEST.COM (short username oryx)
I20260927 21:39:55.618791 7980 webserver.cc:460] Webserver started at http://127.0.202.130:34685/ using document root <none> and password file <none>
I20260927 21:39:55.619616 7980 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:55.619738 7980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:55.626324 7980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:39:55.629877 7995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:55.631588 7980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 21:39:55.631810 7980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "0827c759ab6f4b3fac139ee8d7c53da9"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "aadc7c4a891b51606103af9b1053c091"
server_key_iv: "ce6e42469055dc40ef86fd256a96290c"
server_key_version: "encryptionkey@0"
I20260927 21:39:55.633003 7980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:55.645301 7980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:55.650292 7980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:55.650575 7980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:55.651535 7980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:55.653013 7980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:55.653131 7980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:55.653255 7980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:55.653339 7980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:55.687345 7980 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:45555
I20260927 21:39:55.687402 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:45555 every 8 connection(s)
I20260927 21:39:55.688910 7980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:39:55.695067 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 7980
I20260927 21:39:55.695400 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 7317
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.131@KRBTEST.COM
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.189@KRBTEST.COM
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.188@KRBTEST.COM
Sep 27 21:39:55 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:55.732050 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.696948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.202.131:33469)
0927 21:39:55.705535 (+ 8587us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.705548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.705590 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:39:55.705680 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.705692 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.706045 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.706237 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.711343 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.713062 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.715062 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.715907 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.716032 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.724354 (+ 8322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.724373 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.724392 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.724442 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.728658 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.729501 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.729511 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.729514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.729668 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.730123 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.730128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.730132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.730532 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.730820 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.731281 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.731558 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8405,"thread_start_us":8256,"threads_started":1}
I20260927 21:39:55.732389 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.709765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:33421 (local address 127.0.202.190:32975)
0927 21:39:55.711314 (+ 1549us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.711323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.713130 (+ 1807us) server_negotiation.cc:407] Connection header received
0927 21:39:55.713308 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.713315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.713413 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.713530 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.714997 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.716038 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.718183 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.718779 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.718876 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.724043 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.724051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.724062 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.724104 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.727742 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.728735 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.728740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.728743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.728874 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.729327 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.729331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.729335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.729701 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.729847 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.731385 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.731591 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":1266,"threads_started":1}
I20260927 21:39:55.733217 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.709613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:33421)
0927 21:39:55.712998 (+ 3385us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:55.713021 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:55.713174 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:55.713970 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:55.713975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:55.714517 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:55.714830 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.714840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.716223 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.716229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.718038 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.718049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.719169 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.719176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.719307 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.720146 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:55.720168 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:55.723833 (+ 3665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:55.727903 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.727906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.727920 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.728607 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.728998 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.729003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.729008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.729205 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.729951 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:55.729957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:55.730409 (+ 452us) client_negotiation.cc:769] Sending connection context
0927 21:39:55.730734 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 21:39:55.731289 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3224,"mutex_wait_us":51,"spinlock_wait_cycles":3456,"thread_start_us":3074,"threads_started":1}
I20260927 21:39:55.733511 7440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:55.734175 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.692526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.202.130:46435)
0927 21:39:55.709382 (+ 16856us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:55.709402 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:55.709532 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:55.710552 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:55.710570 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:55.712430 (+ 1860us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:55.713437 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.713448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.714985 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.714990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.717234 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.717248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.718549 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.718557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.718749 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.719653 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:55.719680 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:55.723293 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:55.729512 (+ 6219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.729516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.729519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.730016 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.730445 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.730452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.730455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.730622 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.733118 (+ 2496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:55.733121 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:55.733245 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 21:39:55.733444 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 21:39:55.733724 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16690,"mutex_wait_us":83,"spinlock_wait_cycles":99328,"thread_start_us":15018,"threads_started":1}
I20260927 21:39:55.734596 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.692344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:46435 (local address 127.0.202.188:39799)
0927 21:39:55.694339 (+ 1995us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.694348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.709522 (+ 15174us) server_negotiation.cc:407] Connection header received
0927 21:39:55.709660 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.709666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.709821 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.709932 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.713595 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.714721 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.717433 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.718118 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.718227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.723515 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.723524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.723530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.723574 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.727243 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.730150 (+ 2907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.730155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.730158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.730314 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.730745 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.730750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.730752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.731094 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.731238 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.733847 (+ 2609us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.734098 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1832,"thread_start_us":1736,"threads_started":1}
I20260927 21:39:55.734726 8109 heartbeater.cc:347] Connected to a master server at 127.0.202.190:32975
I20260927 21:39:55.735070 8109 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:55.735414 8108 heartbeater.cc:347] Connected to a master server at 127.0.202.188:39799
I20260927 21:39:55.735539 8108 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:55.735771 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.691950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:36339 (local address 127.0.202.189:43383)
0927 21:39:55.709240 (+ 17290us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:55.709245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:55.709260 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:39:55.709876 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:55.709885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:55.710014 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:55.710154 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:55.713426 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.714616 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.717392 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.718027 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.718134 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.723205 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:55.723212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:55.723217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:55.723259 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:55.727731 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.728764 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.728770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.728773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.728930 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:55.729399 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:55.729406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:55.729409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:55.729752 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:55.729878 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:55.734945 (+ 5067us) server_negotiation.cc:299] Negotiation successful
0927 21:39:55.735259 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17182,"thread_start_us":17069,"threads_started":1}
I20260927 21:39:55.736071 8108 heartbeater.cc:511] Master 127.0.202.188:39799 requested a full tablet report, sending...
I20260927 21:39:55.736455 8109 heartbeater.cc:511] Master 127.0.202.190:32975 requested a full tablet report, sending...
I20260927 21:39:55.737250 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:55.692526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.202.130:36339)
0927 21:39:55.703671 (+ 11145us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:55.703755 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:55.707255 (+ 3500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:55.710554 (+ 3299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:55.710568 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:55.711677 (+ 1109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:55.713228 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.713257 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.714831 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.714836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.717231 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:55.717246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:55.718443 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:55.718450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:55.718740 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:55.719587 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:55.719618 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:55.722967 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:55.727889 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.727902 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.727921 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.728622 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.729081 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:55.729086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:55.729089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:55.729252 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:55.733874 (+ 4622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:55.733879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:55.734050 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 21:39:55.736037 (+ 1987us) client_negotiation.cc:240] Negotiation successful
0927 21:39:55.736501 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10673,"mutex_wait_us":462,"spinlock_wait_cycles":16384,"thread_start_us":16445,"threads_started":1}
I20260927 21:39:55.737787 6988 ts_manager.cc:194] Re-registered known tserver with Master: 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555)
I20260927 21:39:55.738512 8110 heartbeater.cc:347] Connected to a master server at 127.0.202.189:43383
I20260927 21:39:55.738673 8110 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:55.738934 6988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:33421
I20260927 21:39:55.739181 8110 heartbeater.cc:511] Master 127.0.202.189:43383 requested a full tablet report, sending...
I20260927 21:39:55.740000 7784 ts_manager.cc:194] Registered new tserver with Master: 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555)
I20260927 21:39:55.740599 7595 ts_manager.cc:194] Registered new tserver with Master: 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555)
I20260927 21:39:55.834158 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 7317
I20260927 21:39:55.852674 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:33469
--local_ip_for_outbound_sockets=127.0.202.131
--tserver_master_addrs=127.0.202.190:32975
--webserver_port=40995
--webserver_interface=127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab
--principal=oryx/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:39:56.128180 7969 heartbeater.cc:503] Master 127.0.202.190:32975 was elected leader, sending a full tablet report...
W20260927 21:39:56.275524 8121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:39:56.276028 8121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:39:56.276221 8121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:39:56.292472 8121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:39:56.292654 8121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:39:56.292924 8121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:39:56.311153 8121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab
--principal=oryx/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:33469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=40995
--tserver_master_addrs=127.0.202.190:32975,127.0.202.189:43383,127.0.202.188:39799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:39:56.312686 8121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:39:56.314131 8121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:39:56.331264 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:56.331359 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:39:56.331940 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:39:56.332669 8121 server_base.cc:1034] running on GCE node
I20260927 21:39:56.333303 8121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:39:56.334380 8121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:39:56.335875 8121 hybrid_clock.cc:648] HybridClock initialized: now 1790545196335801 us; error 41 us; skew 500 ppm
I20260927 21:39:56.336360 8121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:39:56 dist-test-slave-25dq krb5kdc[6944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:39:56.340178 8121 init.cc:377] Logged in from keytab as oryx/127.0.202.131@KRBTEST.COM (short username oryx)
I20260927 21:39:56.342032 8121 webserver.cc:460] Webserver started at http://127.0.202.131:40995/ using document root <none> and password file <none>
I20260927 21:39:56.342756 8121 fs_manager.cc:359] Metadata directory not provided
I20260927 21:39:56.342916 8121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:39:56.348737 8121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:39:56.351714 8137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:56.353709 8121 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:39:56.353909 8121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "6aa501a5d12a4771ae8ac0bb76970823"
format_stamp: "Formatted at 2026-09-27 21:39:52 on dist-test-slave-25dq"
server_key: "9cdefdd0f5015210c2a73d7bb923ec2d"
server_key_iv: "656c83e56ef161b80973146c8263dcd3"
server_key_version: "encryptionkey@0"
I20260927 21:39:56.355630 8121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:39:56.387353 8121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:39:56.394246 8121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:39:56.394613 8121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:39:56.396056 8121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:39:56.398552 8121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:39:56.398689 8121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:56.398811 8121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:39:56.398905 8121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:39:56.439014 8121 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:33469
I20260927 21:39:56.439062 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:33469 every 8 connection(s)
I20260927 21:39:56.441303 8121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:39:56.445709 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 8121
Sep 27 21:39:56 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:39:56.503090 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.458733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:55593 (local address 127.0.202.190:32975)
0927 21:39:56.473952 (+ 15219us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.473960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.473982 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:39:56.475774 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.475783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.475891 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.475998 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.479705 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.482236 (+ 2531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.485373 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.486055 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.486221 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.492916 (+ 6695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.492926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.492934 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.492990 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.497909 (+ 4919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.499001 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.499007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.499011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.499153 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.499808 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.499849 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.499852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.500245 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.500404 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.502247 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.502530 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15096,"thread_start_us":15255,"threads_started":1}
Sep 27 21:39:56 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.188@KRBTEST.COM
I20260927 21:39:56.506227 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.458714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.131:55593)
0927 21:39:56.473384 (+ 14670us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.473412 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.475216 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.476552 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.476572 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.477834 (+ 1262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:56.479411 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.479443 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.482447 (+ 3004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.482453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.485155 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.485178 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.486616 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.486626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.487041 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.488142 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:56.488193 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:56.492581 (+ 4388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:56.498129 (+ 5548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:56.498139 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:56.498165 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:56.498861 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:56.499293 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:56.499299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:56.499302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:56.499557 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:56.500494 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:56.500503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:56.501008 (+ 505us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.503843 (+ 2835us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.504541 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14479,"mutex_wait_us":98,"thread_start_us":14140,"threads_started":1}
I20260927 21:39:56.508479 8251 heartbeater.cc:347] Connected to a master server at 127.0.202.190:32975
I20260927 21:39:56.508854 8251 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:56.509678 8251 heartbeater.cc:511] Master 127.0.202.190:32975 requested a full tablet report, sending...
I20260927 21:39:56.511673 6988 ts_manager.cc:194] Re-registered known tserver with Master: 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:56.513116 6988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:55593
Sep 27 21:39:56 dist-test-slave-25dq krb5kdc[6944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.189@KRBTEST.COM
I20260927 21:39:56.548094 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.485831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.202.131:50589)
0927 21:39:56.500505 (+ 14674us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.500530 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.500637 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.504025 (+ 3388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.504030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.504615 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:56.504992 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.505010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.510329 (+ 5319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.510347 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.529527 (+ 19180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.529559 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.532603 (+ 3044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.532615 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.532914 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.533773 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:56.533805 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:56.537833 (+ 4028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:56.544045 (+ 6212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:56.544054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:56.544061 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:56.544810 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:56.545419 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:56.545428 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:56.545434 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:56.545640 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:56.546565 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:56.546573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:56.546791 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.546969 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.547402 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14492,"thread_start_us":14862,"threads_started":1}
I20260927 21:39:56.549437 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.486040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:50589 (local address 127.0.202.189:43383)
0927 21:39:56.502144 (+ 16104us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.502156 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.502184 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:39:56.502266 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.502276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.502392 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.502539 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.508455 (+ 5916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.510139 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.529806 (+ 19667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.530792 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.530980 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.538185 (+ 7205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.538198 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.538206 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.538274 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.543811 (+ 5537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.545003 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.545012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.545018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.545230 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.545802 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.545807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.545810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.546265 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.546485 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.548538 (+ 2053us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.548854 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15905,"thread_start_us":16625,"threads_started":1}
I20260927 21:39:56.558900 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.446940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.202.131:36385)
0927 21:39:56.457153 (+ 10213us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.457207 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.475216 (+ 18009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.477167 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.477172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.478414 (+ 1242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:39:56.480104 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.480117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.482447 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.482453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.498637 (+ 16184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.498653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.501014 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.501020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.501676 (+ 656us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.502381 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:39:56.502403 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:39:56.539062 (+ 36659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:39:56.547376 (+ 8314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:56.547384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:56.547391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:56.555595 (+ 8204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:56.556191 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:39:56.556200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:39:56.556206 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:39:56.556443 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:39:56.557336 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:39:56.557344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:39:56.557539 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.557710 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.558290 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9860,"mutex_wait_us":822,"thread_start_us":9584,"threads_started":1}
I20260927 21:39:56.559187 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.446047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:36385 (local address 127.0.202.188:39799)
0927 21:39:56.473005 (+ 26958us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.473012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.473041 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:39:56.475961 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.475968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.476047 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.476138 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.480351 (+ 4213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.481930 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.498866 (+ 16936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.499583 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.499828 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.540938 (+ 41110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.540944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.540947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.540995 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.545620 (+ 4625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.555779 (+ 10159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.555788 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.555795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.556026 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.556598 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.556606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.556612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.557069 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.557244 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.558580 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.558818 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26818,"thread_start_us":27032,"threads_started":1}
I20260927 21:39:56.560082 8250 heartbeater.cc:347] Connected to a master server at 127.0.202.188:39799
I20260927 21:39:56.560248 8250 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:56.560876 8250 heartbeater.cc:511] Master 127.0.202.188:39799 requested a full tablet report, sending...
I20260927 21:39:56.563033 7784 ts_manager.cc:194] Registered new tserver with Master: 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:56.567535 8255 heartbeater.cc:347] Connected to a master server at 127.0.202.189:43383
I20260927 21:39:56.567694 8255 heartbeater.cc:464] Registering TS with master...
I20260927 21:39:56.568065 8255 heartbeater.cc:511] Master 127.0.202.189:43383 requested a full tablet report, sending...
I20260927 21:39:56.569288 7595 ts_manager.cc:194] Registered new tserver with Master: 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:56.571506 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.517669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.0.202.189:43383)
0927 21:39:56.549746 (+ 32077us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.549754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.549776 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:39:56.549834 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.549841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.549930 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.550044 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.553216 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.554769 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.556913 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.557694 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.557824 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.562865 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.562872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.562876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.562967 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.568113 (+ 5146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.569163 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.569168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.569171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.569332 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.569848 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.569853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.569858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.570144 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.570423 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.570846 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.571117 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31932,"thread_start_us":49002,"threads_started":1}
I20260927 21:39:56.576099 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.515535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.202.190:32975)
0927 21:39:56.515752 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.515760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.537957 (+ 22197us) server_negotiation.cc:407] Connection header received
0927 21:39:56.540905 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.540912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.540996 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.541096 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.560448 (+ 19352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.561774 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.563688 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.564445 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.564583 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.569565 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.569573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.569578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.569629 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.573009 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.573884 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.573892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.573899 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.574054 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.574505 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.574510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.574514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.574828 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.574998 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.575450 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.575686 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:39:56.576351 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.506892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.0.202.188:39799)
0927 21:39:56.540531 (+ 33639us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.540537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.540557 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:39:56.540614 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.540619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.540694 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.553255 (+ 12561us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.553560 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.555099 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.564859 (+ 9760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.565536 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.565700 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.569991 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.569999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.570004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.570043 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.573210 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.574053 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.574058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.574062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.574247 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.574670 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.574673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.574676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.575030 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.575257 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.575686 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.575916 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32764,"thread_start_us":43259,"threads_started":1}
I20260927 21:39:56.580538 6988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:56.587846 6988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:56.626562 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.607774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41049 (local address 127.0.0.1:51984)
0927 21:39:56.608962 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.608982 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.609147 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.619330 (+ 10183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.619335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.619359 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:56.619721 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.619729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.621081 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.621088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.623494 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.623513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.625544 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.625552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.625776 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.625817 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.625938 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.626091 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":30,"thread_start_us":351,"threads_started":1}
I20260927 21:39:56.626725 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.608110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.0.202.129:41049)
0927 21:39:56.618854 (+ 10744us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.618865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.618896 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:39:56.618995 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.619001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.619153 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.619356 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:56.619868 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.620941 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.623676 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.625220 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.625362 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.625576 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.626257 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.626440 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10564,"thread_start_us":10455,"threads_started":1}
I20260927 21:39:56.626928 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.610097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.0.202.131:33469)
0927 21:39:56.610769 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.610783 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.616927 (+ 6144us) server_negotiation.cc:407] Connection header received
0927 21:39:56.617107 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.617118 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.617428 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.617630 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:56.618999 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.621186 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.624526 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.625905 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.626020 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.626207 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.626447 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.626609 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":336,"threads_started":1}
I20260927 21:39:56.627422 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.608676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.0.202.130:45555)
0927 21:39:56.617392 (+ 8716us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.617402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.617430 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:39:56.617520 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.617532 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.617767 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.617960 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:56.618900 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.621102 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.623729 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.625220 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.625362 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.625576 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.626944 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.627139 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8551,"thread_start_us":8444,"threads_started":1}
I20260927 21:39:56.627979 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.608363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:45555 (local address 127.0.0.1:45062)
0927 21:39:56.614832 (+ 6469us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.614859 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.615028 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.618037 (+ 3009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.618045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.618081 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:56.618681 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.618696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.621273 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.621278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.623525 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.623539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.625715 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.625720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.625847 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.625869 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.625982 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.626135 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6221,"mutex_wait_us":26,"thread_start_us":339,"threads_started":1}
I20260927 21:39:56.628402 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.609506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:33469 (local address 127.0.0.1:58836)
0927 21:39:56.616797 (+ 7291us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.616813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.616953 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.618161 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.618165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.618194 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:56.618734 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.618747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.622194 (+ 3447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.622199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.624361 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.624373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.626090 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.626095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.626275 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.626306 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.626393 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.626525 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7180,"mutex_wait_us":185,"thread_start_us":7051,"threads_started":1}
I20260927 21:39:56.632676 8043 tablet_service.cc:1467] Processing CreateTablet for tablet bf45e03671904126bb8ed1b35dca7aac (DEFAULT_TABLE table=test-table [id=25b1ba89d34b4af5a3b7da8fe46ad3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:56.634203 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf45e03671904126bb8ed1b35dca7aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:56.635983 8185 tablet_service.cc:1467] Processing CreateTablet for tablet bf45e03671904126bb8ed1b35dca7aac (DEFAULT_TABLE table=test-table [id=25b1ba89d34b4af5a3b7da8fe46ad3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:56.636024 7903 tablet_service.cc:1467] Processing CreateTablet for tablet bf45e03671904126bb8ed1b35dca7aac (DEFAULT_TABLE table=test-table [id=25b1ba89d34b4af5a3b7da8fe46ad3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:56.637679 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf45e03671904126bb8ed1b35dca7aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:56.639153 7903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf45e03671904126bb8ed1b35dca7aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:56.651162 8285 tablet_bootstrap.cc:492] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Bootstrap starting.
I20260927 21:39:56.652937 8286 tablet_bootstrap.cc:492] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Bootstrap starting.
I20260927 21:39:56.655346 8287 tablet_bootstrap.cc:492] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Bootstrap starting.
I20260927 21:39:56.655563 8285 tablet_bootstrap.cc:654] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:56.657146 8286 tablet_bootstrap.cc:654] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:56.658092 8285 log.cc:824] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:56.658346 8286 log.cc:824] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:56.659195 8287 tablet_bootstrap.cc:654] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:56.661000 8287 log.cc:824] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Log is configured to *not* fsync() on all Append() calls
I20260927 21:39:56.661265 8286 tablet_bootstrap.cc:492] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: No bootstrap required, opened a new log
I20260927 21:39:56.661528 8286 ts_tablet_manager.cc:1397] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 21:39:56.662312 8285 tablet_bootstrap.cc:492] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: No bootstrap required, opened a new log
I20260927 21:39:56.662619 8285 ts_tablet_manager.cc:1397] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260927 21:39:56.664388 8287 tablet_bootstrap.cc:492] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: No bootstrap required, opened a new log
I20260927 21:39:56.664664 8287 ts_tablet_manager.cc:1397] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260927 21:39:56.669927 8286 raft_consensus.cc:348] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.670419 8286 raft_consensus.cc:374] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:56.670513 8286 raft_consensus.cc:729] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0827c759ab6f4b3fac139ee8d7c53da9, State: Initialized, Role: FOLLOWER
I20260927 21:39:56.671180 8286 consensus_queue.cc:260] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.673321 8109 heartbeater.cc:503] Master 127.0.202.190:32975 was elected leader, sending a full tablet report...
I20260927 21:39:56.674149 8286 ts_tablet_manager.cc:1428] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260927 21:39:56.673990 8285 raft_consensus.cc:348] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.674454 8285 raft_consensus.cc:374] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:56.674571 8285 raft_consensus.cc:729] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2fa69457ef4c37ab14b56b3f38b197, State: Initialized, Role: FOLLOWER
I20260927 21:39:56.675220 8285 consensus_queue.cc:260] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.677572 8287 raft_consensus.cc:348] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.678129 8287 raft_consensus.cc:374] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:56.678244 8287 raft_consensus.cc:729] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa501a5d12a4771ae8ac0bb76970823, State: Initialized, Role: FOLLOWER
I20260927 21:39:56.679034 8287 consensus_queue.cc:260] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.684224 8285 ts_tablet_manager.cc:1428] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260927 21:39:56.684473 8251 heartbeater.cc:503] Master 127.0.202.190:32975 was elected leader, sending a full tablet report...
I20260927 21:39:56.685073 8287 ts_tablet_manager.cc:1428] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
W20260927 21:39:56.699240 8111 tablet.cc:2369] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:56.764732 8292 raft_consensus.cc:482] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:56.765112 8292 raft_consensus.cc:504] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.766801 8292 leader_election.cc:288] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555), 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:56.774143 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.767172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:56269 (local address 127.0.202.130:45555)
0927 21:39:56.767435 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.767440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.768064 (+ 624us) server_negotiation.cc:407] Connection header received
0927 21:39:56.768343 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.768353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.768438 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.768503 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:56.769316 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.770317 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.772404 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.773214 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.773377 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.773466 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.773750 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.773878 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 21:39:56.774152 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.768337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:57797 (local address 127.0.202.131:33469)
0927 21:39:56.768548 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.768554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.768733 (+ 179us) server_negotiation.cc:407] Connection header received
0927 21:39:56.768955 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.768962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.769038 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.769138 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:56.769780 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.770504 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.772512 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.773296 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.773456 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.773530 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.773815 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.773931 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:39:56.774250 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.767172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:45555 (local address 127.0.202.129:56269)
0927 21:39:56.767931 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.767953 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.768098 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.768696 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.768703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.768732 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:56.769145 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.769156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.770466 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.770471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.772234 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.772246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.773418 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.773424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.773600 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.773625 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.773697 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.773832 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":51,"thread_start_us":409,"threads_started":1}
I20260927 21:39:56.774293 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.768088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:33469 (local address 127.0.202.129:57797)
0927 21:39:56.768651 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:56.768666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:56.768788 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:56.769189 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:56.769193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:56.769250 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:56.769597 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.769611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.770668 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.770672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.772353 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:56.772367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.773525 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.773530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:56.773640 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.773662 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:39:56.773771 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:39:56.773959 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":474,"threads_started":1}
I20260927 21:39:56.775256 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf45e03671904126bb8ed1b35dca7aac" candidate_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" is_pre_election: true
I20260927 21:39:56.775733 8063 raft_consensus.cc:2454] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2fa69457ef4c37ab14b56b3f38b197 in term 0.
I20260927 21:39:56.775671 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf45e03671904126bb8ed1b35dca7aac" candidate_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa501a5d12a4771ae8ac0bb76970823" is_pre_election: true
I20260927 21:39:56.776151 8205 raft_consensus.cc:2454] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2fa69457ef4c37ab14b56b3f38b197 in term 0.
I20260927 21:39:56.776428 7859 leader_election.cc:302] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0827c759ab6f4b3fac139ee8d7c53da9, 1d2fa69457ef4c37ab14b56b3f38b197; no voters:
I20260927 21:39:56.776870 8292 raft_consensus.cc:2781] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:56.777050 8292 raft_consensus.cc:482] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:56.777205 8292 raft_consensus.cc:3037] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:39:56.779160 8261 tablet.cc:2369] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:56.779614 8292 raft_consensus.cc:504] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.780202 8292 leader_election.cc:288] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [CANDIDATE]: Term 1 election: Requested vote from peers 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555), 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:56.780681 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf45e03671904126bb8ed1b35dca7aac" candidate_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0827c759ab6f4b3fac139ee8d7c53da9"
I20260927 21:39:56.780726 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf45e03671904126bb8ed1b35dca7aac" candidate_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa501a5d12a4771ae8ac0bb76970823"
I20260927 21:39:56.780902 8063 raft_consensus.cc:3037] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:56.780907 8205 raft_consensus.cc:3037] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:56.782994 8205 raft_consensus.cc:2454] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2fa69457ef4c37ab14b56b3f38b197 in term 1.
I20260927 21:39:56.783290 8063 raft_consensus.cc:2454] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2fa69457ef4c37ab14b56b3f38b197 in term 1.
I20260927 21:39:56.783363 7859 leader_election.cc:302] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2fa69457ef4c37ab14b56b3f38b197, 6aa501a5d12a4771ae8ac0bb76970823; no voters:
I20260927 21:39:56.783617 8292 raft_consensus.cc:2781] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:56.784106 8292 raft_consensus.cc:686] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 LEADER]: Becoming Leader. State: Replica: 1d2fa69457ef4c37ab14b56b3f38b197, State: Running, Role: LEADER
I20260927 21:39:56.784642 8292 consensus_queue.cc:237] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:56.790642 6988 catalog_manager.cc:5495] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } health_report { overall_health: UNKNOWN } } }
I20260927 21:39:56.813350 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.807536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.202.129:41049)
0927 21:39:56.807750 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.807756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.807777 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:39:56.807996 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.808000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.808076 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.808237 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:56.808690 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.809487 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.811010 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.811445 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.811595 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.812121 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:56.812730 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:56.812896 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.812962 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.813046 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:39:56.821939 8205 raft_consensus.cc:1261] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Refusing update from remote peer 1d2fa69457ef4c37ab14b56b3f38b197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:56.822011 8063 raft_consensus.cc:1261] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 FOLLOWER]: Refusing update from remote peer 1d2fa69457ef4c37ab14b56b3f38b197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 21:39:56.822491 7971 tablet.cc:2369] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:39:56.822991 8292 consensus_queue.cc:1035] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:56.823371 8296 consensus_queue.cc:1035] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:56.837611 8299 mvcc.cc:205] Tried to move back new op lower bound from 7334073126171869184 to 7334073126033281024. Current Snapshot: MvccSnapshot[applied={T|T < 7334073126171869184}]
I20260927 21:39:56.858165 6989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:56.858438 6989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:56.866225 6989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6fe6645ce17499397d98db51fc5732d: Sending DeleteTablet for 3 replicas of tablet bf45e03671904126bb8ed1b35dca7aac
I20260927 21:39:56.867353 7903 tablet_service.cc:1514] Processing DeleteTablet for tablet bf45e03671904126bb8ed1b35dca7aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:56 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:51984
I20260927 21:39:56.867462 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet bf45e03671904126bb8ed1b35dca7aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:56 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:45062
I20260927 21:39:56.868252 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet bf45e03671904126bb8ed1b35dca7aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:56 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:58836
I20260927 21:39:56.869880 8309 tablet_replica.cc:330] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: stopping tablet replica
I20260927 21:39:56.870393 8309 raft_consensus.cc:2229] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:56.872320 8309 raft_consensus.cc:2258] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:56.873310 8310 tablet_replica.cc:330] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: stopping tablet replica
I20260927 21:39:56.873869 8310 raft_consensus.cc:2229] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:56.874482 8310 raft_consensus.cc:2258] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:56.876713 8309 ts_tablet_manager.cc:1905] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:56.876959 8310 ts_tablet_manager.cc:1905] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:56.880476 8311 tablet_replica.cc:330] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: stopping tablet replica
I20260927 21:39:56.881104 8311 raft_consensus.cc:2229] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:56.881721 8311 raft_consensus.cc:2258] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:56.883996 8311 ts_tablet_manager.cc:1905] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:56.885119 8310 ts_tablet_manager.cc:1918] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:56.885308 8310 log.cc:1196] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/bf45e03671904126bb8ed1b35dca7aac
I20260927 21:39:56.885640 8309 ts_tablet_manager.cc:1918] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:56.885859 8309 log.cc:1196] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/bf45e03671904126bb8ed1b35dca7aac
I20260927 21:39:56.886456 8309 ts_tablet_manager.cc:1939] T bf45e03671904126bb8ed1b35dca7aac P 1d2fa69457ef4c37ab14b56b3f38b197: Deleting consensus metadata
I20260927 21:39:56.887250 8310 ts_tablet_manager.cc:1939] T bf45e03671904126bb8ed1b35dca7aac P 0827c759ab6f4b3fac139ee8d7c53da9: Deleting consensus metadata
I20260927 21:39:56.888314 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 6957
I20260927 21:39:56.889137 6975 catalog_manager.cc:4841] TS 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049): tablet bf45e03671904126bb8ed1b35dca7aac (table test-table [id=25b1ba89d34b4af5a3b7da8fe46ad3d0]) successfully deleted
I20260927 21:39:56.889801 7019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:56.899151 8311 ts_tablet_manager.cc:1918] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:56.900025 8311 log.cc:1196] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/bf45e03671904126bb8ed1b35dca7aac
I20260927 21:39:56.901027 8311 ts_tablet_manager.cc:1939] T bf45e03671904126bb8ed1b35dca7aac P 6aa501a5d12a4771ae8ac0bb76970823: Deleting consensus metadata
I20260927 21:39:56.906975 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.871321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.0.202.189:43383)
0927 21:39:56.871556 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.871567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.886444 (+ 14877us) server_negotiation.cc:407] Connection header received
0927 21:39:56.888119 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.888126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.888227 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.888348 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.891710 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.892897 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.894864 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.895536 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.895639 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.899938 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.899947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.899953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.900006 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.903696 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.904797 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.904803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.904807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.904960 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.905382 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.905387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.905391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.905747 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.905981 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.906390 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.906610 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:39:56.924412 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:56.887681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.0.202.188:39799)
0927 21:39:56.887899 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:56.887906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:56.907087 (+ 19181us) server_negotiation.cc:407] Connection header received
0927 21:39:56.907341 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:56.907347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:56.907431 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:56.907502 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:56.909607 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.910782 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.912636 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:56.913295 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:56.913456 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:56.916967 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:56.916975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:56.916980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:56.917031 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:56.921235 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.922076 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.922084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.922089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.922262 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:56.922676 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:56.922682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:56.922686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:56.923047 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:56.923277 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:56.923652 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 21:39:56.923877 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 21:39:57.025288 6976 catalog_manager.cc:4841] TS 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555): tablet bf45e03671904126bb8ed1b35dca7aac (table test-table [id=25b1ba89d34b4af5a3b7da8fe46ad3d0]) successfully deleted
W20260927 21:39:57.025549 6976 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.202.130:45555, user_credentials={real_user=slave}} blocked reactor thread for 135343us
I20260927 21:39:57.025724 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 6957
W20260927 21:39:57.047904 8141 connection.cc:535] server connection from 127.0.0.1:58836 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:39:57.072785 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:57.052392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.0.202.188:39799)
0927 21:39:57.052656 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:57.052664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:57.053748 (+ 1084us) server_negotiation.cc:407] Connection header received
0927 21:39:57.054065 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:57.054074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:57.054183 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:57.054276 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:57.057549 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:57.059137 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:57.061376 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:57.061891 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:57.062048 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:57.066042 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:57.066049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:57.066054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:57.066093 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:57.069763 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:57.070538 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:57.070546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:57.070551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:57.070683 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:57.071088 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:57.071093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:57.071096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:57.071414 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:57.071611 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:57.072187 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 21:39:57.072456 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:39:57.075747 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:57.053492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.0.202.189:43383)
0927 21:39:57.053776 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:57.053784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:57.059338 (+ 5554us) server_negotiation.cc:407] Connection header received
0927 21:39:57.059998 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:57.060007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:57.060106 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:57.060215 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:57.062505 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:57.063579 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:57.065257 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:57.065673 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:57.065812 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:57.068838 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:57.068843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:57.068846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:57.068880 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:57.072091 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:57.073102 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:57.073108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:57.073111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:57.073249 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:57.073819 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:57.073824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:57.073828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:57.074174 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:57.074399 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:57.075178 (+ 779us) server_negotiation.cc:299] Negotiation successful
0927 21:39:57.075401 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:39:57.843195 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:57.842203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.131:42973)
0927 21:39:57.842744 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:57.842935 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":310,"threads_started":1}
W20260927 21:39:57.843585 8251 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:32975 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:57.850982 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:57.850152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:39825)
0927 21:39:57.850628 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:57.850787 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":252,"threads_started":1}
W20260927 21:39:57.851377 8109 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:32975 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:57.852569 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:57.851805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.129:57895)
0927 21:39:57.852240 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:57.852394 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":251,"threads_started":1}
W20260927 21:39:57.853303 7969 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:32975 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:58.365396 8324 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6fe6645ce17499397d98db51fc5732d)
I20260927 21:39:58.365676 8324 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } }
I20260927 21:39:58.367115 8324 leader_election.cc:288] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6fe6645ce17499397d98db51fc5732d (127.0.202.190:32975), 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188:39799)
I20260927 21:39:58.368829 8327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6fe6645ce17499397d98db51fc5732d)
I20260927 21:39:58.369035 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.367267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35048)
0927 21:39:58.368316 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.368548 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":760,"threads_started":1}
I20260927 21:39:58.369148 8327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } }
W20260927 21:39:58.369359 7580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:58.370610 8327 leader_election.cc:288] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6fe6645ce17499397d98db51fc5732d (127.0.202.190:32975), a8e0b28ddf5e489cbc201f40a46d8edd (127.0.202.189:43383)
I20260927 21:39:58.372187 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.371692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35054)
0927 21:39:58.371934 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.372029 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260927 21:39:58.372359 7580 leader_election.cc:334] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6fe6645ce17499397d98db51fc5732d (127.0.202.190:32975): Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:58.373180 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.370725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35050)
0927 21:39:58.372707 (+ 1982us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.372912 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":488,"threads_started":1}
W20260927 21:39:58.374015 7769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:58.376134 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.375505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35060)
0927 21:39:58.375725 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.375921 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:39:58.376446 7769 leader_election.cc:334] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6fe6645ce17499397d98db51fc5732d (127.0.202.190:32975): Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:58.376694 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.367810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.0.202.188:39799)
0927 21:39:58.368581 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.368587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.368611 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:39:58.368690 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.368695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.368783 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.368908 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:58.369760 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.371384 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.374213 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.375615 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.375770 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.375880 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.376136 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.376260 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":533,"threads_started":1}
I20260927 21:39:58.376700 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.367184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:39799 (local address 127.0.0.1:54124)
0927 21:39:58.368160 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.368194 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:58.368367 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:58.368939 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:58.368946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:58.368981 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:58.369353 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.369367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.371576 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.371581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.374071 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.374084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.375792 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.375798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.375982 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.376007 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:39:58.376111 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:39:58.376247 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":4,"thread_start_us":318,"threads_started":1}
I20260927 21:39:58.377708 7794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0bb4317543ea4b71a4b2217d432369aa" is_pre_election: true
I20260927 21:39:58.378053 7794 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8e0b28ddf5e489cbc201f40a46d8edd in term 1.
I20260927 21:39:58.378592 7579 leader_election.cc:302] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bb4317543ea4b71a4b2217d432369aa, a8e0b28ddf5e489cbc201f40a46d8edd; no voters: f6fe6645ce17499397d98db51fc5732d
I20260927 21:39:58.379346 8324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:39:58.379482 8324 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6fe6645ce17499397d98db51fc5732d)
I20260927 21:39:58.379590 8324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:39:58.381142 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.371626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:43383 (local address 127.0.0.1:44046)
0927 21:39:58.373365 (+ 1739us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.373381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:58.373497 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:58.374977 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:58.374982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:58.375011 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:58.375282 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.375293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.376963 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.376968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.378805 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.378821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.380546 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.380553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.380684 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.380710 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:39:58.380816 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:39:58.380932 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"thread_start_us":2060,"threads_started":1}
I20260927 21:39:58.381531 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.371814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.0.202.189:43383)
0927 21:39:58.374662 (+ 2848us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.374667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.374690 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:39:58.374750 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.374756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.374831 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.374925 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:58.375455 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.376835 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.378997 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.380318 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.380465 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.380582 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.380854 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.380990 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2727,"thread_start_us":2567,"threads_started":1}
I20260927 21:39:58.382427 7605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bb4317543ea4b71a4b2217d432369aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" is_pre_election: true
I20260927 21:39:58.382716 8324 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } }
I20260927 21:39:58.383131 7605 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0bb4317543ea4b71a4b2217d432369aa in current term 2: Already voted for candidate a8e0b28ddf5e489cbc201f40a46d8edd in this term.
I20260927 21:39:58.383513 8324 leader_election.cc:288] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [CANDIDATE]: Term 2 election: Requested vote from peers f6fe6645ce17499397d98db51fc5732d (127.0.202.190:32975), 0bb4317543ea4b71a4b2217d432369aa (127.0.202.188:39799)
I20260927 21:39:58.383994 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.383623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35068)
0927 21:39:58.383762 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.383856 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 21:39:58.383919 7794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0bb4317543ea4b71a4b2217d432369aa"
I20260927 21:39:58.384027 7769 leader_election.cc:302] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bb4317543ea4b71a4b2217d432369aa; no voters: a8e0b28ddf5e489cbc201f40a46d8edd, f6fe6645ce17499397d98db51fc5732d
I20260927 21:39:58.384188 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:39:58.385136 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.384693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35084)
0927 21:39:58.384875 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.384979 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 21:39:58.385365 7580 leader_election.cc:334] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6fe6645ce17499397d98db51fc5732d (127.0.202.190:32975): Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
I20260927 21:39:58.386611 7794 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8e0b28ddf5e489cbc201f40a46d8edd in term 2.
I20260927 21:39:58.386834 8327 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:39:58.387064 7579 leader_election.cc:302] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bb4317543ea4b71a4b2217d432369aa, a8e0b28ddf5e489cbc201f40a46d8edd; no voters: f6fe6645ce17499397d98db51fc5732d
I20260927 21:39:58.387367 8324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:39:58.387856 8324 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [term 2 LEADER]: Becoming Leader. State: Replica: a8e0b28ddf5e489cbc201f40a46d8edd, State: Running, Role: LEADER
I20260927 21:39:58.388242 8324 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } }
I20260927 21:39:58.390110 8334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e0b28ddf5e489cbc201f40a46d8edd. Latest consensus state: current_term: 2 leader_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:58.390522 8334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:58.391953 8336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:39:58.395570 8336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:39:58.396453 8336 catalog_manager.cc:1259] Loaded cluster ID: 72bf49c0e83246d8b17dcf907e65b68e
I20260927 21:39:58.396557 8336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:39:58.397770 8336 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:39:58.398562 8336 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Loaded TSK: 0
I20260927 21:39:58.399386 8336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:39:58.420857 7594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:39:58.428376 7594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:39:58.432770 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.432288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35104)
0927 21:39:58.432501 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.432609 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:39:58.432829 7794 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [term 2 FOLLOWER]: Refusing update from remote peer a8e0b28ddf5e489cbc201f40a46d8edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 21:39:58.433648 8334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 21:39:58.433964 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.433541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35110)
0927 21:39:58.433733 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.433815 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 21:39:58.434206 7580 consensus_peers.cc:487] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd -> Peer f6fe6645ce17499397d98db51fc5732d (127.0.202.190:32975): Couldn't send request to peer f6fe6645ce17499397d98db51fc5732d. Status: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:39:58.436169 8327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e0b28ddf5e489cbc201f40a46d8edd. Latest consensus state: current_term: 2 leader_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:58.436515 8327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:58.438354 8334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:58.438861 8338 mvcc.cc:205] Tried to move back new op lower bound from 7334073132773494784 to 7334073132601868288. Current Snapshot: MvccSnapshot[applied={T|T < 7334073132773494784}]
I20260927 21:39:58.439102 8334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd [sys.catalog]: This master's current role is: LEADER
I20260927 21:39:58.439771 8327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f6fe6645ce17499397d98db51fc5732d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 32975 } } peers { permanent_uuid: "a8e0b28ddf5e489cbc201f40a46d8edd" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 43383 } attrs { promote: false } } peers { permanent_uuid: "0bb4317543ea4b71a4b2217d432369aa" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 39799 } attrs { promote: false } } }
I20260927 21:39:58.440024 8327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb4317543ea4b71a4b2217d432369aa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:39:58.458375 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.447846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:45555 (local address 127.0.0.1:45066)
0927 21:39:58.448087 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.448111 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:58.448231 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:58.450032 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:58.450038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:58.450066 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:58.450319 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.450331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.452224 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.452235 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.455057 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.455080 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.457312 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.457322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.457606 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.457666 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:39:58.457750 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:39:58.457955 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 21:39:58.459918 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.448082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.0.202.130:45555)
0927 21:39:58.448568 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.448572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.448625 (+ 53us) server_negotiation.cc:407] Connection header received
0927 21:39:58.449487 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.449498 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.449631 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.450480 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:58.450666 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.452012 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.455341 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.457083 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.459044 (+ 1961us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.459219 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.459283 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.459479 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":298,"threads_started":1}
I20260927 21:39:58.467163 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.448594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41049 (local address 127.0.0.1:51990)
0927 21:39:58.448954 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.448973 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:58.449106 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:58.461035 (+ 11929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:58.461041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:58.461070 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:58.461407 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.461416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.462475 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.462480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.464067 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.464079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.466244 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.466254 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.466528 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.466581 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 21:39:58.466696 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:39:58.466859 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260927 21:39:58.467212 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.449035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.0.202.129:41049)
0927 21:39:58.460716 (+ 11681us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.460720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.460737 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:39:58.460795 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.460799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.460855 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.460982 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:58.461559 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.462229 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.464223 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.465950 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.466122 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.466266 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.466742 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.466945 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11567,"thread_start_us":11458,"threads_started":1}
I20260927 21:39:58.468194 7903 tablet_service.cc:1467] Processing CreateTablet for tablet 6c27d71a428d4e4daa0b1a5681029ed1 (DEFAULT_TABLE table=test-table [id=dbd975730e404ecdac025bef255689ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:58.468478 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 6c27d71a428d4e4daa0b1a5681029ed1 (DEFAULT_TABLE table=test-table [id=dbd975730e404ecdac025bef255689ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:58.468701 7903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c27d71a428d4e4daa0b1a5681029ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:58.468901 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c27d71a428d4e4daa0b1a5681029ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:58.473587 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.449023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.202.131:33469)
0927 21:39:58.467169 (+ 18146us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.467177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.467199 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:39:58.467271 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.467278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.467382 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.467511 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:58.467997 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.469322 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.471525 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.472517 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.472677 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.472790 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.473111 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.473248 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18016,"thread_start_us":17916,"threads_started":1}
I20260927 21:39:58.473599 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.448838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:33469 (local address 127.0.0.1:58850)
0927 21:39:58.458644 (+ 9806us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.458664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:58.458824 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:58.467526 (+ 8702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:58.467531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:58.467573 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:58.467771 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.467782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.469488 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.469494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.471356 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.471368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.472686 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.472691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.472924 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.472950 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:39:58.473074 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 21:39:58.473216 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9717,"mutex_wait_us":37,"thread_start_us":18788,"threads_started":1}
I20260927 21:39:58.475401 8347 tablet_bootstrap.cc:492] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: Bootstrap starting.
I20260927 21:39:58.475771 8185 tablet_service.cc:1467] Processing CreateTablet for tablet 6c27d71a428d4e4daa0b1a5681029ed1 (DEFAULT_TABLE table=test-table [id=dbd975730e404ecdac025bef255689ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:39:58.476228 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c27d71a428d4e4daa0b1a5681029ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:39:58.476783 8346 tablet_bootstrap.cc:492] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: Bootstrap starting.
I20260927 21:39:58.477443 8347 tablet_bootstrap.cc:654] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:58.478503 8346 tablet_bootstrap.cc:654] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:58.479727 8347 tablet_bootstrap.cc:492] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: No bootstrap required, opened a new log
I20260927 21:39:58.479894 8347 ts_tablet_manager.cc:1397] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 21:39:58.480787 8346 tablet_bootstrap.cc:492] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: No bootstrap required, opened a new log
I20260927 21:39:58.480659 8347 raft_consensus.cc:348] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.480926 8346 ts_tablet_manager.cc:1397] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:39:58.480988 8347 raft_consensus.cc:374] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:58.481086 8347 raft_consensus.cc:729] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0827c759ab6f4b3fac139ee8d7c53da9, State: Initialized, Role: FOLLOWER
I20260927 21:39:58.481519 8347 consensus_queue.cc:260] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.481650 8346 raft_consensus.cc:348] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.481886 8346 raft_consensus.cc:374] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:58.481971 8346 raft_consensus.cc:729] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2fa69457ef4c37ab14b56b3f38b197, State: Initialized, Role: FOLLOWER
I20260927 21:39:58.482184 8346 consensus_queue.cc:260] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.483049 8346 ts_tablet_manager.cc:1428] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:39:58.484994 8347 ts_tablet_manager.cc:1428] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 21:39:58.488184 8352 tablet_bootstrap.cc:492] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: Bootstrap starting.
I20260927 21:39:58.489403 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.483728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.129:41019)
0927 21:39:58.489000 (+ 5272us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.489166 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5143,"thread_start_us":5045,"threads_started":1}
I20260927 21:39:58.491462 8352 tablet_bootstrap.cc:654] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: Neither blocks nor log segments found. Creating new log.
I20260927 21:39:58.491541 7970 heartbeater.cc:503] Master 127.0.202.189:43383 was elected leader, sending a full tablet report...
I20260927 21:39:58.492353 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.491585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.129:49441)
0927 21:39:58.492061 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.492161 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
I20260927 21:39:58.493907 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.485572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:34233)
0927 21:39:58.493477 (+ 7905us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.493663 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7745,"thread_start_us":7615,"threads_started":1}
I20260927 21:39:58.494935 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.494614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:42923)
0927 21:39:58.494764 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.494831 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:39:58.495004 8352 tablet_bootstrap.cc:492] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: No bootstrap required, opened a new log
I20260927 21:39:58.495040 8110 heartbeater.cc:503] Master 127.0.202.189:43383 was elected leader, sending a full tablet report...
I20260927 21:39:58.495162 8352 ts_tablet_manager.cc:1397] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 21:39:58.496008 8352 raft_consensus.cc:348] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.496245 8352 raft_consensus.cc:374] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:39:58.496367 8352 raft_consensus.cc:729] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa501a5d12a4771ae8ac0bb76970823, State: Initialized, Role: FOLLOWER
I20260927 21:39:58.496601 8352 consensus_queue.cc:260] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.497686 8352 ts_tablet_manager.cc:1428] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:39:58.498670 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.497742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.131:54787)
0927 21:39:58.498290 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.498428 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":628,"threads_started":1}
I20260927 21:39:58.499718 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.499301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.131:40533)
0927 21:39:58.499483 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.499565 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:39:58.500165 8255 heartbeater.cc:503] Master 127.0.202.189:43383 was elected leader, sending a full tablet report...
I20260927 21:39:58.516865 8350 raft_consensus.cc:482] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:39:58.517088 8350 raft_consensus.cc:504] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.518716 8350 leader_election.cc:288] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049), 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:58.524207 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.519004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:37665 (local address 127.0.202.129:41049)
0927 21:39:58.519265 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.519271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.519291 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:39:58.519437 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.519443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.519565 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.519657 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:58.520162 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.520892 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.522512 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.523266 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.523448 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.523522 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.523831 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.523955 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 21:39:58.524230 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.518872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41049 (local address 127.0.202.130:37665)
0927 21:39:58.519086 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.519107 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:58.519280 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:58.519709 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:58.519713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:58.519738 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:58.520022 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.520030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.521028 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.521035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.522360 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.522371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.523460 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.523469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.523672 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.523695 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:39:58.523797 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:39:58.523931 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260927 21:39:58.525132 7923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c27d71a428d4e4daa0b1a5681029ed1" candidate_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" is_pre_election: true
I20260927 21:39:58.525636 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.519375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:38471 (local address 127.0.202.131:33469)
0927 21:39:58.519621 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.519629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.520165 (+ 536us) server_negotiation.cc:407] Connection header received
0927 21:39:58.520392 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.520397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.520466 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.520569 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:39:58.521099 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.521843 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.523677 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.524679 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.524892 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.524977 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.525237 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.525378 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:39:58.525661 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.519333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:33469 (local address 127.0.202.130:38471)
0927 21:39:58.520056 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.520072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:39:58.520228 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:39:58.520583 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:39:58.520587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:39:58.520611 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:39:58.520935 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.520945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.521990 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.521997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.523494 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:39:58.523506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.524942 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.524948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:39:58.525084 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.525106 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:39:58.525177 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:39:58.525323 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":1,"thread_start_us":395,"threads_started":1}
I20260927 21:39:58.525628 7923 raft_consensus.cc:2454] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0827c759ab6f4b3fac139ee8d7c53da9 in term 0.
I20260927 21:39:58.526273 7998 leader_election.cc:302] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0827c759ab6f4b3fac139ee8d7c53da9, 1d2fa69457ef4c37ab14b56b3f38b197; no voters:
I20260927 21:39:58.526377 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c27d71a428d4e4daa0b1a5681029ed1" candidate_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa501a5d12a4771ae8ac0bb76970823" is_pre_election: true
I20260927 21:39:58.526628 8205 raft_consensus.cc:2454] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0827c759ab6f4b3fac139ee8d7c53da9 in term 0.
I20260927 21:39:58.526708 8350 raft_consensus.cc:2781] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:39:58.526836 8350 raft_consensus.cc:482] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:39:58.526927 8350 raft_consensus.cc:3037] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:58.529428 8350 raft_consensus.cc:504] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.530030 8350 leader_election.cc:288] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [CANDIDATE]: Term 1 election: Requested vote from peers 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049), 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469)
I20260927 21:39:58.530599 7923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c27d71a428d4e4daa0b1a5681029ed1" candidate_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2fa69457ef4c37ab14b56b3f38b197"
I20260927 21:39:58.530608 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c27d71a428d4e4daa0b1a5681029ed1" candidate_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa501a5d12a4771ae8ac0bb76970823"
I20260927 21:39:58.530809 8205 raft_consensus.cc:3037] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:58.530812 7923 raft_consensus.cc:3037] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:39:58.533151 7923 raft_consensus.cc:2454] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0827c759ab6f4b3fac139ee8d7c53da9 in term 1.
I20260927 21:39:58.533212 8205 raft_consensus.cc:2454] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0827c759ab6f4b3fac139ee8d7c53da9 in term 1.
I20260927 21:39:58.533565 7998 leader_election.cc:302] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0827c759ab6f4b3fac139ee8d7c53da9, 1d2fa69457ef4c37ab14b56b3f38b197; no voters:
I20260927 21:39:58.533862 8350 raft_consensus.cc:2781] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:39:58.534387 8350 raft_consensus.cc:686] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 LEADER]: Becoming Leader. State: Replica: 0827c759ab6f4b3fac139ee8d7c53da9, State: Running, Role: LEADER
I20260927 21:39:58.534816 8350 consensus_queue.cc:237] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } }
I20260927 21:39:58.535485 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.535057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:48901)
0927 21:39:58.535264 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.535357 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 21:39:58.538383 7594 catalog_manager.cc:5495] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130). New cstate: current_term: 1 leader_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0827c759ab6f4b3fac139ee8d7c53da9" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 45555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 } health_report { overall_health: UNKNOWN } } }
I20260927 21:39:58.559408 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.554197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.202.130:45555)
0927 21:39:58.554441 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.554446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.554461 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:39:58.554544 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.554548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.554623 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.554779 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:39:58.555149 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.555848 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.557231 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.557683 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.557830 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.558234 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:39:58.558822 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:39:58.559000 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.559056 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.559138 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:39:58.563755 8205 raft_consensus.cc:1261] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Refusing update from remote peer 0827c759ab6f4b3fac139ee8d7c53da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:58.564407 7923 raft_consensus.cc:1261] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 FOLLOWER]: Refusing update from remote peer 0827c759ab6f4b3fac139ee8d7c53da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:39:58.564833 8359 consensus_queue.cc:1035] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aa501a5d12a4771ae8ac0bb76970823" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:58.565315 8350 consensus_queue.cc:1035] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d2fa69457ef4c37ab14b56b3f38b197" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:39:58.568161 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.567469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.131:35273)
0927 21:39:58.567785 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.567925 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 21:39:58.568221 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.567650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.129:56041)
0927 21:39:58.567828 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.567996 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:39:58.575327 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.574845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.129:48491)
0927 21:39:58.575089 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.575179 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260927 21:39:58.584147 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.575282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:40475)
0927 21:39:58.583179 (+ 7897us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.583320 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7723}
I20260927 21:39:58.588037 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.587467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.130:40291)
0927 21:39:58.587719 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.587850 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 21:39:58.589316 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.588808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.202.131:39609)
0927 21:39:58.589025 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.589115 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 21:39:58.592207 7595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:58.592552 7595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:39:58.598937 7595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8e0b28ddf5e489cbc201f40a46d8edd: Sending DeleteTablet for 3 replicas of tablet 6c27d71a428d4e4daa0b1a5681029ed1
I20260927 21:39:58.599887 7903 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c27d71a428d4e4daa0b1a5681029ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:58 UTC) from {username='oryx', principal='oryx/127.0.202.189@KRBTEST.COM'} at 127.0.0.1:51990
I20260927 21:39:58.600033 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c27d71a428d4e4daa0b1a5681029ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:58 UTC) from {username='oryx', principal='oryx/127.0.202.189@KRBTEST.COM'} at 127.0.0.1:58850
I20260927 21:39:58.600615 8372 tablet_replica.cc:330] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: stopping tablet replica
I20260927 21:39:58.600829 8372 raft_consensus.cc:2229] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:58.600826 8184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c27d71a428d4e4daa0b1a5681029ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:58 UTC) from {username='oryx', principal='oryx/127.0.202.189@KRBTEST.COM'} at 127.0.0.1:58850
I20260927 21:39:58.601063 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c27d71a428d4e4daa0b1a5681029ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:39:58 UTC) from {username='oryx', principal='oryx/127.0.202.189@KRBTEST.COM'} at 127.0.0.1:45066
I20260927 21:39:58.601138 8372 raft_consensus.cc:2258] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:58.602515 8372 ts_tablet_manager.cc:1905] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:58.602684 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 7762
I20260927 21:39:58.611738 8372 ts_tablet_manager.cc:1918] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:58.611923 8372 log.cc:1196] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/6c27d71a428d4e4daa0b1a5681029ed1
I20260927 21:39:58.612491 8372 ts_tablet_manager.cc:1939] T 6c27d71a428d4e4daa0b1a5681029ed1 P 1d2fa69457ef4c37ab14b56b3f38b197: Deleting consensus metadata
I20260927 21:39:58.613951 7581 catalog_manager.cc:4841] TS 1d2fa69457ef4c37ab14b56b3f38b197 (127.0.202.129:41049): tablet 6c27d71a428d4e4daa0b1a5681029ed1 (table test-table [id=dbd975730e404ecdac025bef255689ac]) successfully deleted
I20260927 21:39:58.616860 8374 tablet_replica.cc:330] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: stopping tablet replica
I20260927 21:39:58.617131 8374 raft_consensus.cc:2229] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:39:58.617921 8374 raft_consensus.cc:2258] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:58.618623 8373 tablet_replica.cc:330] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: stopping tablet replica
I20260927 21:39:58.618834 8373 raft_consensus.cc:2229] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:39:58.619117 8373 raft_consensus.cc:2258] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:39:58.620811 8373 ts_tablet_manager.cc:1905] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:58.621243 8374 ts_tablet_manager.cc:1905] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:39:58.627800 8373 ts_tablet_manager.cc:1918] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:58.627954 8373 log.cc:1196] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/6c27d71a428d4e4daa0b1a5681029ed1
I20260927 21:39:58.628468 8373 ts_tablet_manager.cc:1939] T 6c27d71a428d4e4daa0b1a5681029ed1 P 6aa501a5d12a4771ae8ac0bb76970823: Deleting consensus metadata
I20260927 21:39:58.629819 7582 catalog_manager.cc:4841] TS 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469): tablet 6c27d71a428d4e4daa0b1a5681029ed1 (table test-table [id=dbd975730e404ecdac025bef255689ac]) successfully deleted
W20260927 21:39:58.630863 7582 catalog_manager.cc:4808] TS 6aa501a5d12a4771ae8ac0bb76970823 (127.0.202.131:33469): delete failed for tablet 6c27d71a428d4e4daa0b1a5681029ed1 because the tablet was not found. No further retry: Not found: Tablet not found: 6c27d71a428d4e4daa0b1a5681029ed1
I20260927 21:39:58.631445 8374 ts_tablet_manager.cc:1918] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:39:58.631712 8374 log.cc:1196] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/6c27d71a428d4e4daa0b1a5681029ed1
I20260927 21:39:58.632670 8374 ts_tablet_manager.cc:1939] T 6c27d71a428d4e4daa0b1a5681029ed1 P 0827c759ab6f4b3fac139ee8d7c53da9: Deleting consensus metadata
I20260927 21:39:58.634246 7582 catalog_manager.cc:4841] TS 0827c759ab6f4b3fac139ee8d7c53da9 (127.0.202.130:45555): tablet 6c27d71a428d4e4daa0b1a5681029ed1 (table test-table [id=dbd975730e404ecdac025bef255689ac]) successfully deleted
I20260927 21:39:58.637580 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.603421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.202.129:41049)
0927 21:39:58.603919 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.603928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.619785 (+ 15857us) server_negotiation.cc:407] Connection header received
0927 21:39:58.620163 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.620169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.620256 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.620460 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:58.622226 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.623284 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.625109 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.625748 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.625849 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.629766 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:58.629787 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:58.629811 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:58.629857 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:58.633945 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:58.635071 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:58.635101 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:58.635104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:58.635266 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:58.635663 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:58.635668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:58.635672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:58.636133 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:58.636470 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.636991 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.637225 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260927 21:39:58.639671 7963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:58.766163 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 7762
I20260927 21:39:58.798193 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 7980
I20260927 21:39:58.815064 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.799020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.0.202.130:45555)
0927 21:39:58.799228 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.799235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.799294 (+ 59us) server_negotiation.cc:407] Connection header received
0927 21:39:58.799468 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.799474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.799578 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.799696 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:58.801246 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.802392 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.803860 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.804657 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.804790 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.807519 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:58.807534 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:58.807556 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:58.807624 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:58.812053 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:58.812869 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:58.812886 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:58.812890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:58.813066 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:58.813425 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:58.813430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:58.813434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:58.813899 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:58.814216 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.814435 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.814658 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:39:58.815935 8103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:58.895022 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.894490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35114)
0927 21:39:58.894683 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.894780 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 21:39:58.896045 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.895668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:32975 (local address 127.0.0.1:35122)
0927 21:39:58.895860 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:39:58.895927 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:32975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:39:58.939397 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 7980
I20260927 21:39:58.959235 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 8121
I20260927 21:39:58.974434 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:39:58.960007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.0.202.131:33469)
0927 21:39:58.960192 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:39:58.960197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:39:58.960215 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:39:58.960380 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:39:58.960384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:39:58.960456 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:39:58.960541 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:39:58.962090 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.963012 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.964676 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:39:58.965186 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:39:58.965300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:39:58.967849 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:39:58.967864 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:39:58.967882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:39:58.967989 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:39:58.971377 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:58.972082 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:58.972093 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:58.972097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:58.972234 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:39:58.972751 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:39:58.972754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:39:58.972757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:39:58.973124 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:39:58.973441 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 21:39:58.973827 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 21:39:58.974083 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:39:58.975211 8245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:59.071682 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 8121
I20260927 21:39:59.089073 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 7450
I20260927 21:39:59.090236 7625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:59.214552 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 7450
I20260927 21:39:59.236531 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 7546
I20260927 21:39:59.237670 7814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:39:59.353708 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 7546
2026-09-27T21:39:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10021 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:39:59 dist-test-slave-25dq krb5kdc[8383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:39:59 dist-test-slave-25dq krb5kdc[8383](info): set up 2 sockets
Sep 27 21:39:59 dist-test-slave-25dq krb5kdc[8383](info): commencing operation
krb5kdc: starting...
W20260927 21:40:00.510116 810 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.010s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:40:00 dist-test-slave-25dq krb5kdc[8383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:40:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab.
Entry for principal oryx/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab.
I20260927 21:40:00.746598 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44379
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab
--principal=oryx/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:01.101343 8396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:01.101643 8396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:01.101732 8396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:01.109311 8396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:40:01.109436 8396 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:40:01.109477 8396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:01.109530 8396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:01.109572 8396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:40:01.109635 8396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:40:01.120062 8396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:44379
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.190.keytab
--principal=oryx/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:01.121409 8396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:01.122735 8396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:01.135057 8402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:01.135159 8404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:01.135830 8401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:01.136055 8396 server_base.cc:1034] running on GCE node
I20260927 21:40:01.137013 8396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:01.138525 8396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:01.139804 8396 hybrid_clock.cc:648] HybridClock initialized: now 1790545201139726 us; error 50 us; skew 500 ppm
I20260927 21:40:01.140173 8396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:01 dist-test-slave-25dq krb5kdc[8383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545201, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:01.143484 8396 init.cc:377] Logged in from keytab as oryx/127.0.202.190@KRBTEST.COM (short username oryx)
I20260927 21:40:01.144990 8396 webserver.cc:460] Webserver started at http://127.0.202.190:43929/ using document root <none> and password file <none>
I20260927 21:40:01.145674 8396 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:01.145759 8396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:01.146050 8396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:01.149492 8396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "df33ef9edcfc4d169c15a316ae2dc261"
format_stamp: "Formatted at 2026-09-27 21:40:01 on dist-test-slave-25dq"
server_key: "12c7f027cb6f1519c6a866ed1f5739bf"
server_key_iv: "81c8e6733901f6d3a25a596840bf0a24"
server_key_version: "encryptionkey@0"
I20260927 21:40:01.150355 8396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "df33ef9edcfc4d169c15a316ae2dc261"
format_stamp: "Formatted at 2026-09-27 21:40:01 on dist-test-slave-25dq"
server_key: "12c7f027cb6f1519c6a866ed1f5739bf"
server_key_iv: "81c8e6733901f6d3a25a596840bf0a24"
server_key_version: "encryptionkey@0"
I20260927 21:40:01.156188 8396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:40:01.159979 8411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:01.161782 8396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:01.161949 8396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "df33ef9edcfc4d169c15a316ae2dc261"
format_stamp: "Formatted at 2026-09-27 21:40:01 on dist-test-slave-25dq"
server_key: "12c7f027cb6f1519c6a866ed1f5739bf"
server_key_iv: "81c8e6733901f6d3a25a596840bf0a24"
server_key_version: "encryptionkey@0"
I20260927 21:40:01.162164 8396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:01.179382 8396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:01.183974 8396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:01.184219 8396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:01.207729 8396 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:44379
I20260927 21:40:01.207749 8472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:44379 every 8 connection(s)
I20260927 21:40:01.209219 8396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:40:01.209909 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 8396
I20260927 21:40:01.210139 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:40:01.210945 810 external_mini_cluster.cc:1428] Setting key 38edda0de1453f33ec824cc7357d1395
I20260927 21:40:01.214576 8473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:40:01 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:40:01.225392 8473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Bootstrap starting.
I20260927 21:40:01.228991 8473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:01.230041 8473 log.cc:824] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:01.233297 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:01.213206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.0.202.190:44379)
0927 21:40:01.214023 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:01.214035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:01.214076 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:40:01.215752 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:01.215773 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:01.216157 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:01.216627 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:01.218061 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.219412 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.220706 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.221373 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.221566 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:01.225669 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:01.225677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:01.225699 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:01.225743 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:01.228882 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:01.229650 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:01.229657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:01.229664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:01.229823 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:01.230202 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:01.230209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:01.230216 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:01.231034 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:01.231448 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:01.231780 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 21:40:01.232191 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":379,"threads_started":1}
I20260927 21:40:01.233430 8473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: No bootstrap required, opened a new log
I20260927 21:40:01.241684 8473 raft_consensus.cc:348] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33ef9edcfc4d169c15a316ae2dc261" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44379 } }
I20260927 21:40:01.242139 8473 raft_consensus.cc:374] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:01.242255 8473 raft_consensus.cc:729] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df33ef9edcfc4d169c15a316ae2dc261, State: Initialized, Role: FOLLOWER
I20260927 21:40:01.243073 8473 consensus_queue.cc:260] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33ef9edcfc4d169c15a316ae2dc261" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44379 } }
I20260927 21:40:01.243396 8473 raft_consensus.cc:388] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:40:01.243551 8473 raft_consensus.cc:482] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:40:01.243772 8473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:01.246268 8473 raft_consensus.cc:504] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33ef9edcfc4d169c15a316ae2dc261" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44379 } }
I20260927 21:40:01.246853 8473 leader_election.cc:302] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df33ef9edcfc4d169c15a316ae2dc261; no voters:
I20260927 21:40:01.247470 8473 leader_election.cc:288] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:40:01.247670 8478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:01.248572 8478 raft_consensus.cc:686] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [term 1 LEADER]: Becoming Leader. State: Replica: df33ef9edcfc4d169c15a316ae2dc261, State: Running, Role: LEADER
I20260927 21:40:01.249155 8478 consensus_queue.cc:237] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33ef9edcfc4d169c15a316ae2dc261" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44379 } }
I20260927 21:40:01.249646 8473 sys_catalog.cc:564] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:40:01.252082 8480 sys_catalog.cc:455] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df33ef9edcfc4d169c15a316ae2dc261. Latest consensus state: current_term: 1 leader_uuid: "df33ef9edcfc4d169c15a316ae2dc261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33ef9edcfc4d169c15a316ae2dc261" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44379 } } }
I20260927 21:40:01.252542 8480 sys_catalog.cc:458] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:01.253163 8479 sys_catalog.cc:455] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df33ef9edcfc4d169c15a316ae2dc261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33ef9edcfc4d169c15a316ae2dc261" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44379 } } }
I20260927 21:40:01.253518 8479 sys_catalog.cc:458] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:01.253680 8487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:40:01.259383 8487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:40:01.269209 8487 catalog_manager.cc:1347] Generated new cluster ID: 93cc2b5205fd4474a4bf66e335d55ca4
I20260927 21:40:01.269418 8487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:40:01.287321 8487 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:40:01.288954 8487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:40:01.297842 8487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Generated new TSK 0
I20260927 21:40:01.298987 8487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab.
Entry for principal oryx/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab.
I20260927 21:40:01.360178 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44379
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab
--principal=oryx/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:01.703791 8499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:01.704113 8499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:01.704280 8499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:01.711962 8499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:01.712082 8499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:01.712229 8499 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:40:01.712375 8499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:40:01.722427 8499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab
--principal=oryx/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:01.723770 8499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:01.725193 8499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:01.739333 8507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:01.739617 8504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:01.739646 8505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:01.740571 8499 server_base.cc:1034] running on GCE node
I20260927 21:40:01.741233 8499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:01.742267 8499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:01.743589 8499 hybrid_clock.cc:648] HybridClock initialized: now 1790545201743478 us; error 53 us; skew 500 ppm
I20260927 21:40:01.743965 8499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:01 dist-test-slave-25dq krb5kdc[8383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545201, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:01.747967 8499 init.cc:377] Logged in from keytab as oryx/127.0.202.129@KRBTEST.COM (short username oryx)
I20260927 21:40:01.749965 8499 webserver.cc:460] Webserver started at http://127.0.202.129:35479/ using document root <none> and password file <none>
I20260927 21:40:01.750752 8499 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:01.750900 8499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:01.751346 8499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:01.755064 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "39d8991f8c67403091da063700f0e381"
format_stamp: "Formatted at 2026-09-27 21:40:01 on dist-test-slave-25dq"
server_key: "2e2c51dfb466a689173c608fba9fdf16"
server_key_iv: "7e533d9fe62049e18e55048abbb1159e"
server_key_version: "encryptionkey@0"
I20260927 21:40:01.755967 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "39d8991f8c67403091da063700f0e381"
format_stamp: "Formatted at 2026-09-27 21:40:01 on dist-test-slave-25dq"
server_key: "2e2c51dfb466a689173c608fba9fdf16"
server_key_iv: "7e533d9fe62049e18e55048abbb1159e"
server_key_version: "encryptionkey@0"
I20260927 21:40:01.762197 8499 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:40:01.766322 8514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:01.767951 8499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:40:01.768218 8499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "39d8991f8c67403091da063700f0e381"
format_stamp: "Formatted at 2026-09-27 21:40:01 on dist-test-slave-25dq"
server_key: "2e2c51dfb466a689173c608fba9fdf16"
server_key_iv: "7e533d9fe62049e18e55048abbb1159e"
server_key_version: "encryptionkey@0"
I20260927 21:40:01.768579 8499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:01.786677 8499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:01.793376 8499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:01.793764 8499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:01.797431 8499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:01.797554 8499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:01.797724 8499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:01.797807 8499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:40:01 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545201, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:40:01.825471 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:01.797920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:37243 (local address 127.0.202.190:44379)
0927 21:40:01.798844 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:01.798849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:01.800355 (+ 1506us) server_negotiation.cc:407] Connection header received
0927 21:40:01.803428 (+ 3073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:01.803438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:01.803563 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:01.803660 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:01.806843 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.808103 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.810510 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.811324 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.811502 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:01.818115 (+ 6613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:01.818121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:01.818124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:01.818163 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:01.821119 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:01.822307 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:01.822314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:01.822319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:01.822481 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:01.822982 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:01.822987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:01.822991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:01.823407 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:01.823578 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:01.825029 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0927 21:40:01.825222 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":731,"threads_started":1}
I20260927 21:40:01.827934 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:01.798949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44379 (local address 127.0.202.129:37243)
0927 21:40:01.800110 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:01.800158 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:01.802988 (+ 2830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:01.804342 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:01.804356 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:01.805363 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:01.806583 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:01.806618 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.808325 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.808331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:01.810297 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:01.810322 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.811804 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.811815 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:01.812207 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:01.813228 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:01.813256 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:01.817770 (+ 4514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:01.821304 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:01.821314 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:01.821335 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:01.822081 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:01.822599 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:01.822604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:01.822608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:01.822844 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:01.823584 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:01.823598 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:01.824390 (+ 792us) client_negotiation.cc:769] Sending connection context
0927 21:40:01.824895 (+ 505us) client_negotiation.cc:240] Negotiation successful
0927 21:40:01.825745 (+ 850us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":559,"threads_started":1}
I20260927 21:40:01.847747 8499 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:32809
I20260927 21:40:01.847937 8634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:32809 every 8 connection(s)
I20260927 21:40:01.849814 8499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:40:01.852715 8635 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44379
I20260927 21:40:01.852985 8635 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:01.853902 8635 heartbeater.cc:511] Master 127.0.202.190:44379 requested a full tablet report, sending...
I20260927 21:40:01.854327 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 8499
I20260927 21:40:01.854537 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:40:01.855340 810 external_mini_cluster.cc:1428] Setting key 04067bf59e4c8ca33d164aa590b5f53c
I20260927 21:40:01.855352 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:01.835707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44379 (local address 127.0.202.129:37551)
0927 21:40:01.836559 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:01.836605 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:01.836846 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:01.837299 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:01.837303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:01.838293 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:01.838689 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:01.838701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.840131 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.840137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:01.841895 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:01.841912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.843473 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.843485 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:01.843928 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:01.845047 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:01.845073 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:01.846248 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:01.851510 (+ 5262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:01.851518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:01.851567 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:01.852185 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:01.852932 (+ 747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:01.852939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:01.852945 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:01.853192 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:01.853971 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:01.853979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:01.854303 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 21:40:01.854476 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 21:40:01.854859 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":452,"threads_started":1}
I20260927 21:40:01.856431 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:01.835713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:37551 (local address 127.0.202.190:44379)
0927 21:40:01.835943 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:01.835952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:01.836767 (+ 815us) server_negotiation.cc:407] Connection header received
0927 21:40:01.837031 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:01.837035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:01.837105 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:01.837261 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:01.838871 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.839964 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.842067 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:01.842834 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:01.843030 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:01.846562 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:01.846573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:01.846580 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:01.846638 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:01.851287 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:01.852358 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:01.852365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:01.852369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:01.852517 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:01.853352 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:01.853357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:01.853360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:01.853801 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:01.853968 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:01.855700 (+ 1732us) server_negotiation.cc:299] Negotiation successful
0927 21:40:01.855946 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:40:01.856630 8428 ts_manager.cc:194] Registered new tserver with Master: 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809)
I20260927 21:40:01.858929 8428 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:37243
WARNING: no policy specified for oryx/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab.
Entry for principal oryx/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab.
I20260927 21:40:01.912953 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44379
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab
--principal=oryx/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:02.253991 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:02.254282 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:02.254381 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:02.261945 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:02.262070 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:02.262156 8639 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:40:02.262228 8639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:40:02.273037 8639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab
--principal=oryx/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:02.274757 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:02.276227 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:02.290198 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:02.290237 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:02.290266 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:02.291127 8639 server_base.cc:1034] running on GCE node
I20260927 21:40:02.291976 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:02.293207 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:02.294493 8639 hybrid_clock.cc:648] HybridClock initialized: now 1790545202294422 us; error 46 us; skew 500 ppm
I20260927 21:40:02.294898 8639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:02 dist-test-slave-25dq krb5kdc[8383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:02.298422 8639 init.cc:377] Logged in from keytab as oryx/127.0.202.130@KRBTEST.COM (short username oryx)
I20260927 21:40:02.299921 8639 webserver.cc:460] Webserver started at http://127.0.202.130:33039/ using document root <none> and password file <none>
I20260927 21:40:02.300724 8639 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:02.300812 8639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:02.301136 8639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:02.304266 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "17c503e92ef443b991e0e33272258a97"
format_stamp: "Formatted at 2026-09-27 21:40:02 on dist-test-slave-25dq"
server_key: "37809abff3b4e46ef7da1760e0c5ab06"
server_key_iv: "714d5cc7d26f67b8207a0f468187fea8"
server_key_version: "encryptionkey@0"
I20260927 21:40:02.305172 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "17c503e92ef443b991e0e33272258a97"
format_stamp: "Formatted at 2026-09-27 21:40:02 on dist-test-slave-25dq"
server_key: "37809abff3b4e46ef7da1760e0c5ab06"
server_key_iv: "714d5cc7d26f67b8207a0f468187fea8"
server_key_version: "encryptionkey@0"
I20260927 21:40:02.310576 8639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 21:40:02.315366 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:02.317049 8639 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 21:40:02.317263 8639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "17c503e92ef443b991e0e33272258a97"
format_stamp: "Formatted at 2026-09-27 21:40:02 on dist-test-slave-25dq"
server_key: "37809abff3b4e46ef7da1760e0c5ab06"
server_key_iv: "714d5cc7d26f67b8207a0f468187fea8"
server_key_version: "encryptionkey@0"
I20260927 21:40:02.317507 8639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:02.339596 8639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:02.344473 8639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:02.344785 8639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:02.347538 8639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:02.347651 8639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:02.347812 8639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:02.347889 8639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:40:02 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:40:02.374694 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.348405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:44265 (local address 127.0.202.190:44379)
0927 21:40:02.348606 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:02.348612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:02.350070 (+ 1458us) server_negotiation.cc:407] Connection header received
0927 21:40:02.353305 (+ 3235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:02.353312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:02.353432 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:02.353566 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:02.355790 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.357107 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.359695 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.360482 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.360636 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.366004 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:02.366014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:02.366023 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:02.366071 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:02.369707 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.371109 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.371117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.371122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.371278 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.371943 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.371952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.371956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.372414 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:02.372590 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:02.373748 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0927 21:40:02.374107 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:40:02.375363 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.348969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44379 (local address 127.0.202.130:44265)
0927 21:40:02.349858 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:02.349943 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:02.352899 (+ 2956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:02.353845 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:02.353853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:02.354671 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:02.355519 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.355569 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.357341 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.357347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.359486 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.359503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.360761 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.360769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.361012 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.361894 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:02.361925 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:02.365644 (+ 3719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:02.369942 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.369959 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.369985 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.370923 (+ 938us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.371427 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.371432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.371436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.371769 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.372571 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:02.372579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:02.373180 (+ 601us) client_negotiation.cc:769] Sending connection context
0927 21:40:02.373624 (+ 444us) client_negotiation.cc:240] Negotiation successful
0927 21:40:02.374093 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":296,"threads_started":1}
I20260927 21:40:02.401057 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.381182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:53689 (local address 127.0.202.190:44379)
0927 21:40:02.381391 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:02.381396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:02.382233 (+ 837us) server_negotiation.cc:407] Connection header received
0927 21:40:02.382441 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:02.382447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:02.382528 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:02.382642 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:02.383622 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.385002 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.387593 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.388364 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.388533 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.392122 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:02.392138 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:02.392144 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:02.392206 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:02.396990 (+ 4784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.398266 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.398274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.398278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.398509 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.399087 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.399097 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.399104 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.399725 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:02.399923 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:02.400448 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 21:40:02.400696 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:40:02.401160 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.381409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44379 (local address 127.0.202.130:53689)
0927 21:40:02.382037 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:02.382056 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:02.382307 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:02.382664 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:02.382669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:02.383156 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:02.383441 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.383451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.385230 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.385235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.387139 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.387159 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.388610 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.388623 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.388836 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.390207 (+ 1371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:02.390233 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:02.391394 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:02.397275 (+ 5881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.397284 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.397290 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.397974 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.398682 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.398686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.398688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.398941 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.399921 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:02.399926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:02.400170 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 21:40:02.400365 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 21:40:02.400717 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":370,"threads_started":1}
I20260927 21:40:02.410589 8639 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:40887
I20260927 21:40:02.410622 8773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:40887 every 8 connection(s)
I20260927 21:40:02.411811 8639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:40:02.413990 8774 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44379
I20260927 21:40:02.414253 8774 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:02.415048 8774 heartbeater.cc:511] Master 127.0.202.190:44379 requested a full tablet report, sending...
I20260927 21:40:02.416586 8428 ts_manager.cc:194] Registered new tserver with Master: 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887)
I20260927 21:40:02.417464 8428 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:44265
I20260927 21:40:02.418608 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 8639
I20260927 21:40:02.418783 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:40:02.419364 810 external_mini_cluster.cc:1428] Setting key 1daab095d99ece44ddf03d4acaef812c
WARNING: no policy specified for oryx/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab.
Entry for principal oryx/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab.
I20260927 21:40:02.474108 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44379
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab
--principal=oryx/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:02.813190 8778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:02.813491 8778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:02.813635 8778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:02.821367 8778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:02.821482 8778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:02.821672 8778 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:40:02.821789 8778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:40:02.831959 8778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab
--principal=oryx/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.202.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:02.833303 8778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:02.834684 8778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:02.850674 8786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:02.850757 8784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:02.851344 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:02.852033 8778 server_base.cc:1034] running on GCE node
I20260927 21:40:02.852837 8778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:02.853935 8778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:02.855279 8778 hybrid_clock.cc:648] HybridClock initialized: now 1790545202855217 us; error 32 us; skew 500 ppm
I20260927 21:40:02.855667 8778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:02 dist-test-slave-25dq krb5kdc[8383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:02.859658 8778 init.cc:377] Logged in from keytab as oryx/127.0.202.131@KRBTEST.COM (short username oryx)
I20260927 21:40:02.861526 8778 webserver.cc:460] Webserver started at http://127.0.202.131:46847/ using document root <none> and password file <none>
I20260927 21:40:02.862299 8778 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:02.862442 8778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:02.862798 8778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:02.863179 8635 heartbeater.cc:503] Master 127.0.202.190:44379 was elected leader, sending a full tablet report...
I20260927 21:40:02.866390 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "8ddb4a7a6e574e6bab24f2128ff2f306"
format_stamp: "Formatted at 2026-09-27 21:40:02 on dist-test-slave-25dq"
server_key: "0b53ec5e600543de791bf49e6afd2474"
server_key_iv: "e8896fbe0a4011ac946b38c119a11e86"
server_key_version: "encryptionkey@0"
I20260927 21:40:02.867368 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "8ddb4a7a6e574e6bab24f2128ff2f306"
format_stamp: "Formatted at 2026-09-27 21:40:02 on dist-test-slave-25dq"
server_key: "0b53ec5e600543de791bf49e6afd2474"
server_key_iv: "e8896fbe0a4011ac946b38c119a11e86"
server_key_version: "encryptionkey@0"
I20260927 21:40:02.873565 8778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:40:02.877723 8793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:02.879355 8778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:40:02.879618 8778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "8ddb4a7a6e574e6bab24f2128ff2f306"
format_stamp: "Formatted at 2026-09-27 21:40:02 on dist-test-slave-25dq"
server_key: "0b53ec5e600543de791bf49e6afd2474"
server_key_iv: "e8896fbe0a4011ac946b38c119a11e86"
server_key_version: "encryptionkey@0"
I20260927 21:40:02.879931 8778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:02.902791 8778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:02.907788 8778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:02.908110 8778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:02.911042 8778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:02.911176 8778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:02.911319 8778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:02.911391 8778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:40:02 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:40:02.937819 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.911779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:43551 (local address 127.0.202.190:44379)
0927 21:40:02.912423 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:02.912428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:02.913692 (+ 1264us) server_negotiation.cc:407] Connection header received
0927 21:40:02.915974 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:02.915980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:02.916077 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:02.916232 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:02.919583 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.920670 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.923474 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.924093 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.924250 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.928981 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:02.928990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:02.928996 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:02.929044 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:02.932988 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.934389 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.934398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.934402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.934583 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.935342 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.935348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.935351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.935770 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:02.935962 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:02.936914 (+ 952us) server_negotiation.cc:299] Negotiation successful
0927 21:40:02.937220 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":383,"threads_started":1}
I20260927 21:40:02.938690 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.912613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44379 (local address 127.0.202.131:43551)
0927 21:40:02.913509 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:02.913558 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:02.915518 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:02.916895 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:02.916913 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:02.918044 (+ 1131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:02.919315 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.919359 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.920930 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.920938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.923249 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.923271 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.924418 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.924428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.924657 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.925380 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:02.925408 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:02.928678 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:02.933213 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.933227 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.933256 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.934183 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.934736 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.934745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.934749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.935149 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.935919 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:02.935929 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:02.936430 (+ 501us) client_negotiation.cc:769] Sending connection context
0927 21:40:02.936790 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 21:40:02.937363 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":339,"threads_started":1}
I20260927 21:40:02.963158 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.946386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:57439 (local address 127.0.202.190:44379)
0927 21:40:02.946612 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:02.946621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:02.947449 (+ 828us) server_negotiation.cc:407] Connection header received
0927 21:40:02.947764 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:02.947768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:02.947849 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:02.947943 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:02.949707 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.950659 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.953203 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.953814 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.953963 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.957033 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:02.957038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:02.957041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:02.957075 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:02.960023 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.960903 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.960909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.960912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.961099 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:02.961542 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:02.961547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:02.961550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:02.961981 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:02.962141 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:02.962531 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 21:40:02.962766 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:40:02.963281 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:02.946500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44379 (local address 127.0.202.131:57439)
0927 21:40:02.947307 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:02.947334 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:02.947598 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:02.948015 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:02.948022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:02.948829 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:02.949322 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.949341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.950832 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.950840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.953018 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:02.953037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:02.954228 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:02.954238 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:02.954502 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:02.955409 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:02.955427 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:02.956813 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:02.960173 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.960179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.960183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.960703 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.961233 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:02.961239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:02.961243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:02.961417 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:02.962095 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:02.962099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:02.962283 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 21:40:02.962434 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 21:40:02.962789 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":367,"threads_started":1}
I20260927 21:40:02.972085 8778 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:41375
I20260927 21:40:02.972121 8913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:41375 every 8 connection(s)
I20260927 21:40:02.973649 8778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:40:02.976387 8914 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44379
I20260927 21:40:02.976632 8914 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:02.977452 8914 heartbeater.cc:511] Master 127.0.202.190:44379 requested a full tablet report, sending...
I20260927 21:40:02.979156 8428 ts_manager.cc:194] Registered new tserver with Master: 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:02.980063 8428 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:43551
I20260927 21:40:02.980588 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 8778
I20260927 21:40:02.980784 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:40:02.981316 810 external_mini_cluster.cc:1428] Setting key 2179c6744a2f69f45331deb440d70e5e
I20260927 21:40:02.984715 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:40:03 dist-test-slave-25dq krb5kdc[8383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:40:03 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:03 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:03.420593 8774 heartbeater.cc:503] Master 127.0.202.190:44379 was elected leader, sending a full tablet report...
I20260927 21:40:03.983214 8914 heartbeater.cc:503] Master 127.0.202.190:44379 was elected leader, sending a full tablet report...
W20260927 21:40:04.144090 8922 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.107s user 0.005s sys 0.010s
W20260927 21:40:04.145195 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:03.030035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.0.202.190:44379)
0927 21:40:03.030435 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:03.030443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:03.030707 (+ 264us) server_negotiation.cc:407] Connection header received
0927 21:40:03.030944 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:03.030949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:03.031013 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:03.031152 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:03.032595 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:03.033299 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:03.035258 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:03.035681 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:03.035821 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.144983 (+1109162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49272: BlockingRecv error: recv got EOF from 127.0.0.1:49272 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 21:40:04 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:40:04.163414 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.148972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.0.202.190:44379)
0927 21:40:04.149172 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.149181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.149552 (+ 371us) server_negotiation.cc:407] Connection header received
0927 21:40:04.149771 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.149776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.149879 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.149985 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:04.151355 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.152198 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.153541 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.153967 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.154113 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.158003 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:04.158008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:04.158012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:04.158046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:04.160835 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:04.161510 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:04.161518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:04.161526 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:04.161648 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:04.162015 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:04.162022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:04.162024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:04.162294 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:04.162579 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.162857 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.163097 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:40:04.168053 8428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:40:04.175312 8428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:40:04.202001 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.189249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.202.131:41375)
0927 21:40:04.189866 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.189880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.191327 (+ 1447us) server_negotiation.cc:407] Connection header received
0927 21:40:04.191948 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.191959 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.192254 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.192476 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:04.193626 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.195354 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.198699 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.200361 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.200487 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.200729 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.201473 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.201659 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
I20260927 21:40:04.203047 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.189045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41375 (local address 127.0.0.1:35020)
0927 21:40:04.191208 (+ 2163us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.191235 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.191442 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.192803 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.192811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.192844 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:04.193393 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.193406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.195538 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.195544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.198512 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.198528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.200922 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.200929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.201130 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.201201 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.202320 (+ 1119us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.202540 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2022,"mutex_wait_us":76,"thread_start_us":2225,"threads_started":1}
I20260927 21:40:04.204388 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.188680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:40887 (local address 127.0.0.1:46540)
0927 21:40:04.190499 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.190552 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.190744 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.193170 (+ 2426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.193178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.193210 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:04.193664 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.193678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.197076 (+ 3398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.197088 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.200620 (+ 3532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.200634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.203000 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.203009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.203364 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.203415 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.203489 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.203649 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1559,"mutex_wait_us":1,"thread_start_us":1390,"threads_started":1}
I20260927 21:40:04.206066 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.189032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.0.202.130:40887)
0927 21:40:04.191357 (+ 2325us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.191368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.191392 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:40:04.192663 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.192674 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.192944 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.193882 (+ 938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:04.194088 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.196901 (+ 2813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.200831 (+ 3930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.202725 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.204102 (+ 1377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.205481 (+ 1379us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.205564 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.205751 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2151,"thread_start_us":2018,"threads_started":1}
I20260927 21:40:04.208072 8844 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffa0c2fc1f14285b77172a36e84c6ff (DEFAULT_TABLE table=test-table [id=39f2c9f57d144f48ae5b8c3d86af3457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:04.209822 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffa0c2fc1f14285b77172a36e84c6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.210350 8704 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffa0c2fc1f14285b77172a36e84c6ff (DEFAULT_TABLE table=test-table [id=39f2c9f57d144f48ae5b8c3d86af3457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:04.212419 8704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffa0c2fc1f14285b77172a36e84c6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.213557 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.191467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:32809 (local address 127.0.0.1:43022)
0927 21:40:04.205073 (+ 13606us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.205182 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.205306 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.206073 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.206079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.206111 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:04.206490 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.206499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.208173 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.208178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.210450 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.210466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.212647 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.212654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.212829 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.212864 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.212952 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.213189 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13379,"thread_start_us":13206,"threads_started":1}
I20260927 21:40:04.214395 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.191892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.0.202.129:32809)
0927 21:40:04.205395 (+ 13503us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.205405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.205438 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:40:04.205554 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.205566 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.205860 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.206070 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:04.206679 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.207992 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.210662 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.212144 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.212284 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.212506 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.213831 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.214014 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13319,"thread_start_us":13892,"threads_started":1}
I20260927 21:40:04.218142 8565 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffa0c2fc1f14285b77172a36e84c6ff (DEFAULT_TABLE table=test-table [id=39f2c9f57d144f48ae5b8c3d86af3457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:04.221220 8565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffa0c2fc1f14285b77172a36e84c6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.232877 8939 tablet_bootstrap.cc:492] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Bootstrap starting.
I20260927 21:40:04.234334 8940 tablet_bootstrap.cc:492] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Bootstrap starting.
I20260927 21:40:04.237300 8939 tablet_bootstrap.cc:654] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.238445 8940 tablet_bootstrap.cc:654] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.238436 8939 log.cc:824] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:04.239809 8940 log.cc:824] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:04.240860 8942 tablet_bootstrap.cc:492] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Bootstrap starting.
I20260927 21:40:04.242349 8939 tablet_bootstrap.cc:492] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: No bootstrap required, opened a new log
I20260927 21:40:04.242640 8939 ts_tablet_manager.cc:1397] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260927 21:40:04.243083 8940 tablet_bootstrap.cc:492] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: No bootstrap required, opened a new log
I20260927 21:40:04.243343 8940 ts_tablet_manager.cc:1397] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 21:40:04.244655 8942 tablet_bootstrap.cc:654] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.245760 8942 log.cc:824] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:04.252565 8942 tablet_bootstrap.cc:492] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: No bootstrap required, opened a new log
I20260927 21:40:04.253801 8942 ts_tablet_manager.cc:1397] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260927 21:40:04.253950 8939 raft_consensus.cc:348] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.254453 8939 raft_consensus.cc:374] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.254566 8939 raft_consensus.cc:729] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.254268 8940 raft_consensus.cc:348] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.254719 8940 raft_consensus.cc:374] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.254812 8940 raft_consensus.cc:729] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c503e92ef443b991e0e33272258a97, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.255256 8939 consensus_queue.cc:260] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.255465 8940 consensus_queue.cc:260] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.257642 8940 ts_tablet_manager.cc:1428] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260927 21:40:04.259150 8939 ts_tablet_manager.cc:1428] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260927 21:40:04.265779 8942 raft_consensus.cc:348] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.266201 8942 raft_consensus.cc:374] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.266307 8942 raft_consensus.cc:729] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d8991f8c67403091da063700f0e381, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.266848 8942 consensus_queue.cc:260] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.268177 8942 ts_tablet_manager.cc:1428] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Time spent starting tablet: real 0.014s user 0.008s sys 0.003s
I20260927 21:40:04.336897 8945 raft_consensus.cc:482] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:04.337188 8945 raft_consensus.cc:504] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.338932 8945 leader_election.cc:288] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809)
I20260927 21:40:04.345688 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.339326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:56321 (local address 127.0.202.129:32809)
0927 21:40:04.339700 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.339709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.340051 (+ 342us) server_negotiation.cc:407] Connection header received
0927 21:40:04.340278 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.340282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.340412 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.340507 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:04.341036 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.341795 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.343957 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.344764 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.344951 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.345023 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.345294 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.345420 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260927 21:40:04.345850 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.339203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:32809 (local address 127.0.202.131:56321)
0927 21:40:04.339933 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.339950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.340131 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.340536 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.340541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.340568 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:04.340884 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.340892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.341917 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.341926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.343720 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.343733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.344968 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.344974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.345140 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.345163 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.345270 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.345409 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":2,"thread_start_us":373,"threads_started":1}
I20260927 21:40:04.346540 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.339205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:40887 (local address 127.0.202.131:47319)
0927 21:40:04.340277 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.340299 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.340493 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.340900 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.340904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.340951 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:04.341284 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.341293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.342406 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.342412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.344429 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.344441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.345632 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.345639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.345846 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.345887 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.346016 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.346168 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":126,"thread_start_us":503,"threads_started":1}
I20260927 21:40:04.346558 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.339398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:47319 (local address 127.0.202.130:40887)
0927 21:40:04.339616 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.339623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.340431 (+ 808us) server_negotiation.cc:407] Connection header received
0927 21:40:04.340655 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.340659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.340776 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.340881 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:04.341452 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.342263 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.344585 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.345396 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.345593 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.345682 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.346180 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.346319 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:40:04.346771 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa0c2fc1f14285b77172a36e84c6ff" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381" is_pre_election: true
I20260927 21:40:04.347308 8585 raft_consensus.cc:2454] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 0.
I20260927 21:40:04.347743 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa0c2fc1f14285b77172a36e84c6ff" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97" is_pre_election: true
I20260927 21:40:04.348012 8794 leader_election.cc:302] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:04.348340 8729 raft_consensus.cc:2454] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 0.
I20260927 21:40:04.348587 8945 raft_consensus.cc:2781] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:04.348760 8945 raft_consensus.cc:482] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:04.348883 8945 raft_consensus.cc:3037] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:04.351382 8945 raft_consensus.cc:504] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.351953 8945 leader_election.cc:288] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 election: Requested vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809)
I20260927 21:40:04.352514 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa0c2fc1f14285b77172a36e84c6ff" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97"
I20260927 21:40:04.352524 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffa0c2fc1f14285b77172a36e84c6ff" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381"
I20260927 21:40:04.352753 8585 raft_consensus.cc:3037] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:04.352764 8729 raft_consensus.cc:3037] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:40:04.353703 8636 tablet.cc:2369] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:40:04.355219 8585 raft_consensus.cc:2454] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 1.
I20260927 21:40:04.355500 8729 raft_consensus.cc:2454] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 1.
I20260927 21:40:04.355643 8794 leader_election.cc:302] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:04.355943 8945 raft_consensus.cc:2781] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:04.356393 8945 raft_consensus.cc:686] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 LEADER]: Becoming Leader. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Running, Role: LEADER
I20260927 21:40:04.356875 8945 consensus_queue.cc:237] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } }
I20260927 21:40:04.361733 8428 catalog_manager.cc:5495] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20260927 21:40:04.395097 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.389189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.0.202.131:41375)
0927 21:40:04.389399 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.389406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.389514 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:40:04.389801 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.389806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.389909 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.390075 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:04.390497 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.391284 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.392824 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.393271 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.393420 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.393889 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:04.394455 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:04.394650 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.394715 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.394801 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:40:04.403592 8729 raft_consensus.cc:1261] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb4a7a6e574e6bab24f2128ff2f306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:04.403592 8585 raft_consensus.cc:1261] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb4a7a6e574e6bab24f2128ff2f306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:04.404605 8950 consensus_queue.cc:1035] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:04.404949 8945 consensus_queue.cc:1035] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:40:04.415319 8775 tablet.cc:2369] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:40:04.416977 8953 mvcc.cc:205] Tried to move back new op lower bound from 7334073157227806720 to 7334073157049356288. Current Snapshot: MvccSnapshot[applied={T|T < 7334073157227806720}]
I20260927 21:40:04.440026 8427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:04.440527 8427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:04.444310 8427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Sending DeleteTablet for 3 replicas of tablet 4ffa0c2fc1f14285b77172a36e84c6ff
I20260927 21:40:04.445643 8844 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ffa0c2fc1f14285b77172a36e84c6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:04 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:35020
I20260927 21:40:04.445833 8704 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ffa0c2fc1f14285b77172a36e84c6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:04 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:46540
I20260927 21:40:04.446079 8565 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ffa0c2fc1f14285b77172a36e84c6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:04 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:43022
I20260927 21:40:04.452675 8427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:40:04.454654 8427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:40:04.468730 8843 tablet_service.cc:1467] Processing CreateTablet for tablet 5680539a19954d3680f4e4e496cfd3ca (DEFAULT_TABLE table=test-table [id=1c91bc4e7a9540878f70c505bcfcdb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:04.469105 8564 tablet_service.cc:1467] Processing CreateTablet for tablet 5680539a19954d3680f4e4e496cfd3ca (DEFAULT_TABLE table=test-table [id=1c91bc4e7a9540878f70c505bcfcdb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:04.469333 8843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5680539a19954d3680f4e4e496cfd3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.469636 8564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5680539a19954d3680f4e4e496cfd3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.475108 8965 tablet_replica.cc:330] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: stopping tablet replica
I20260927 21:40:04.475703 8965 raft_consensus.cc:2229] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:04.478127 8939 tablet_bootstrap.cc:492] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Bootstrap starting.
I20260927 21:40:04.478428 8965 raft_consensus.cc:2258] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:04.479323 8964 tablet_replica.cc:330] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: stopping tablet replica
I20260927 21:40:04.479727 8964 raft_consensus.cc:2229] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:04.479930 8939 tablet_bootstrap.cc:654] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.480271 8964 raft_consensus.cc:2258] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:04.481704 8963 tablet_replica.cc:330] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: stopping tablet replica
I20260927 21:40:04.482930 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 5680539a19954d3680f4e4e496cfd3ca (DEFAULT_TABLE table=test-table [id=1c91bc4e7a9540878f70c505bcfcdb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:04.483383 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5680539a19954d3680f4e4e496cfd3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.483572 8965 ts_tablet_manager.cc:1905] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:04.486366 8964 ts_tablet_manager.cc:1905] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:04.485145 8963 raft_consensus.cc:2229] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:40:04.487216 8963 raft_consensus.cc:2258] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:04.492017 8942 tablet_bootstrap.cc:492] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: Bootstrap starting.
I20260927 21:40:04.493857 8939 tablet_bootstrap.cc:492] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: No bootstrap required, opened a new log
I20260927 21:40:04.494019 8939 ts_tablet_manager.cc:1397] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260927 21:40:04.494835 8964 ts_tablet_manager.cc:1918] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:04.495023 8964 log.cc:1196] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/4ffa0c2fc1f14285b77172a36e84c6ff
I20260927 21:40:04.494958 8939 raft_consensus.cc:348] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.495206 8939 raft_consensus.cc:374] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.495304 8939 raft_consensus.cc:729] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.495615 8964 ts_tablet_manager.cc:1939] T 4ffa0c2fc1f14285b77172a36e84c6ff P 17c503e92ef443b991e0e33272258a97: Deleting consensus metadata
I20260927 21:40:04.495599 8939 consensus_queue.cc:260] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.495903 8965 ts_tablet_manager.cc:1918] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:04.496129 8965 log.cc:1196] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/4ffa0c2fc1f14285b77172a36e84c6ff
I20260927 21:40:04.496750 8965 ts_tablet_manager.cc:1939] T 4ffa0c2fc1f14285b77172a36e84c6ff P 39d8991f8c67403091da063700f0e381: Deleting consensus metadata
I20260927 21:40:04.496824 8942 tablet_bootstrap.cc:654] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.497579 8413 catalog_manager.cc:4841] TS 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887): tablet 4ffa0c2fc1f14285b77172a36e84c6ff (table test-table [id=39f2c9f57d144f48ae5b8c3d86af3457]) successfully deleted
I20260927 21:40:04.498613 8939 ts_tablet_manager.cc:1428] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:40:04.499231 8940 tablet_bootstrap.cc:492] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: Bootstrap starting.
I20260927 21:40:04.500908 8940 tablet_bootstrap.cc:654] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.502357 8963 ts_tablet_manager.cc:1905] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:04.502398 8412 catalog_manager.cc:4841] TS 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809): tablet 4ffa0c2fc1f14285b77172a36e84c6ff (table test-table [id=39f2c9f57d144f48ae5b8c3d86af3457]) successfully deleted
I20260927 21:40:04.502852 8942 tablet_bootstrap.cc:492] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: No bootstrap required, opened a new log
I20260927 21:40:04.502985 8942 ts_tablet_manager.cc:1397] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260927 21:40:04.503791 8942 raft_consensus.cc:348] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.504009 8942 raft_consensus.cc:374] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.504106 8942 raft_consensus.cc:729] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d8991f8c67403091da063700f0e381, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.504398 8942 consensus_queue.cc:260] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.506515 8942 ts_tablet_manager.cc:1428] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 21:40:04.509193 8963 ts_tablet_manager.cc:1918] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:04.509378 8963 log.cc:1196] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/4ffa0c2fc1f14285b77172a36e84c6ff
I20260927 21:40:04.509971 8963 ts_tablet_manager.cc:1939] T 4ffa0c2fc1f14285b77172a36e84c6ff P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting consensus metadata
I20260927 21:40:04.511870 8415 catalog_manager.cc:4841] TS 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375): tablet 4ffa0c2fc1f14285b77172a36e84c6ff (table test-table [id=39f2c9f57d144f48ae5b8c3d86af3457]) successfully deleted
I20260927 21:40:04.514015 8940 tablet_bootstrap.cc:492] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: No bootstrap required, opened a new log
I20260927 21:40:04.514178 8940 ts_tablet_manager.cc:1397] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260927 21:40:04.515008 8940 raft_consensus.cc:348] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.515256 8940 raft_consensus.cc:374] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.515345 8940 raft_consensus.cc:729] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c503e92ef443b991e0e33272258a97, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.515614 8940 consensus_queue.cc:260] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.516396 8940 ts_tablet_manager.cc:1428] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260927 21:40:04.732352 8915 tablet.cc:2369] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:40:04.770399 8947 raft_consensus.cc:482] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:04.770668 8947 raft_consensus.cc:504] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.772835 8947 leader_election.cc:288] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:04.783267 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.773338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:34911 (local address 127.0.202.130:40887)
0927 21:40:04.773569 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.773578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.776152 (+ 2574us) server_negotiation.cc:407] Connection header received
0927 21:40:04.776330 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.776336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.776413 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.776515 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:04.777273 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.778140 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.780980 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.782260 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.782451 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.782543 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.782839 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.782968 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:40:04.783541 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.773861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:40887 (local address 127.0.202.129:34911)
0927 21:40:04.776021 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.776038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.776160 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.776595 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.776601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.776628 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:04.777108 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.777117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.778263 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.778271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.780806 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.780822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.782508 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.782512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.782641 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.782669 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.782815 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.782954 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2031,"mutex_wait_us":82,"thread_start_us":1772,"threads_started":1}
I20260927 21:40:04.784366 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5680539a19954d3680f4e4e496cfd3ca" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97" is_pre_election: true
I20260927 21:40:04.784650 8729 raft_consensus.cc:2454] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 0.
I20260927 21:40:04.785290 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.773589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:41835 (local address 127.0.202.131:41375)
0927 21:40:04.773779 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.773786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.775676 (+ 1890us) server_negotiation.cc:407] Connection header received
0927 21:40:04.776017 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.776023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.776120 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.776230 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:04.777055 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.778298 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.782629 (+ 4331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.784165 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.784379 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.784470 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.784830 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.784980 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:40:04.785190 8516 leader_election.cc:302] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c503e92ef443b991e0e33272258a97, 39d8991f8c67403091da063700f0e381; no voters:
I20260927 21:40:04.785646 8947 raft_consensus.cc:2781] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:04.785801 8947 raft_consensus.cc:482] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:04.785904 8947 raft_consensus.cc:3037] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:04.786254 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.773300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41375 (local address 127.0.202.129:41835)
0927 21:40:04.775567 (+ 2267us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.775589 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.775799 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.776324 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.776328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.776349 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:04.776788 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.776808 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.778440 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.778444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.780806 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.780822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.784397 (+ 3575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.784404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.784578 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.784611 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.785478 (+ 867us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.785625 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2125,"mutex_wait_us":82,"thread_start_us":1956,"threads_started":1}
I20260927 21:40:04.789145 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5680539a19954d3680f4e4e496cfd3ca" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" is_pre_election: true
I20260927 21:40:04.789248 8947 raft_consensus.cc:504] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.789614 8868 raft_consensus.cc:2454] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 0.
I20260927 21:40:04.790176 8947 leader_election.cc:288] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 election: Requested vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:04.790472 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5680539a19954d3680f4e4e496cfd3ca" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97"
I20260927 21:40:04.790700 8729 raft_consensus.cc:3037] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:04.791137 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5680539a19954d3680f4e4e496cfd3ca" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306"
I20260927 21:40:04.791364 8868 raft_consensus.cc:3037] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:04.793874 8729 raft_consensus.cc:2454] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 1.
I20260927 21:40:04.794315 8516 leader_election.cc:302] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c503e92ef443b991e0e33272258a97, 39d8991f8c67403091da063700f0e381; no voters:
I20260927 21:40:04.794652 8947 raft_consensus.cc:2781] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:04.794791 8868 raft_consensus.cc:2454] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 1.
I20260927 21:40:04.796598 8947 raft_consensus.cc:686] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 1 LEADER]: Becoming Leader. State: Replica: 39d8991f8c67403091da063700f0e381, State: Running, Role: LEADER
I20260927 21:40:04.797049 8947 consensus_queue.cc:237] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.801280 8427 catalog_manager.cc:5495] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39d8991f8c67403091da063700f0e381 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:40:04 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545201, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.190@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:40:04.897046 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.870920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44379 (local address 127.0.0.1:49292)
0927 21:40:04.871785 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:04.871806 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:04.871970 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:04.872422 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:04.872427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:04.873341 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:04.873630 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.873645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.874896 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.874902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.876493 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:04.876508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.877370 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.877377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:04.877565 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.886954 (+ 9389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:04.886990 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:04.890195 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:04.894089 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:04.894105 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:04.894117 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:04.894607 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:04.895062 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:04.895068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:04.895072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:04.895244 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:04.895807 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:04.895816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:04.895993 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 21:40:04.896183 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 21:40:04.896523 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":551,"threads_started":1}
I20260927 21:40:04.897073 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:04.870923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.0.202.190:44379)
0927 21:40:04.871670 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:04.871677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:04.871928 (+ 251us) server_negotiation.cc:407] Connection header received
0927 21:40:04.872137 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:04.872144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:04.872211 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:04.872376 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:04.873909 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.874736 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.876650 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:04.877168 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:04.877307 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:04.890513 (+ 13206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:04.890519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:04.890523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:04.890559 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:04.893818 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:04.894797 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:04.894803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:04.894807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:04.894944 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:04.895372 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:04.895377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:04.895383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:04.895666 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:04.895850 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:04.896277 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 21:40:04.896686 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":523,"thread_start_us":316,"threads_started":1}
I20260927 21:40:04.902740 8427 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:49292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:40:04.903898 8427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:40:04.910910 8564 tablet_service.cc:1467] Processing CreateTablet for tablet d7411778aab44bf1809fb24affb448b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8da0fe83263147b09d9069c642c66902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:40:04.910910 8703 tablet_service.cc:1467] Processing CreateTablet for tablet d7411778aab44bf1809fb24affb448b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8da0fe83263147b09d9069c642c66902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:40:04.911031 8844 tablet_service.cc:1467] Processing CreateTablet for tablet d7411778aab44bf1809fb24affb448b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8da0fe83263147b09d9069c642c66902]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:40:04.911470 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7411778aab44bf1809fb24affb448b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.911474 8564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7411778aab44bf1809fb24affb448b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.911552 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7411778aab44bf1809fb24affb448b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:04.917325 8940 tablet_bootstrap.cc:492] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: Bootstrap starting.
I20260927 21:40:04.918632 8940 tablet_bootstrap.cc:654] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.919363 8939 tablet_bootstrap.cc:492] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: Bootstrap starting.
I20260927 21:40:04.919898 8942 tablet_bootstrap.cc:492] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: Bootstrap starting.
I20260927 21:40:04.921037 8939 tablet_bootstrap.cc:654] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.921597 8942 tablet_bootstrap.cc:654] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:04.922189 8940 tablet_bootstrap.cc:492] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: No bootstrap required, opened a new log
I20260927 21:40:04.922377 8940 ts_tablet_manager.cc:1397] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 21:40:04.923270 8939 tablet_bootstrap.cc:492] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: No bootstrap required, opened a new log
I20260927 21:40:04.923259 8940 raft_consensus.cc:348] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.923450 8939 ts_tablet_manager.cc:1397] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:40:04.923542 8940 raft_consensus.cc:374] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.923637 8940 raft_consensus.cc:729] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c503e92ef443b991e0e33272258a97, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.924016 8942 tablet_bootstrap.cc:492] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: No bootstrap required, opened a new log
I20260927 21:40:04.923899 8940 consensus_queue.cc:260] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.924122 8939 raft_consensus.cc:348] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.924389 8939 raft_consensus.cc:374] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.924476 8939 raft_consensus.cc:729] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.924607 8940 ts_tablet_manager.cc:1428] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:40:04.924402 8946 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:04.924813 8946 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:40:04.924749 8939 consensus_queue.cc:260] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.925268 8939 ts_tablet_manager.cc:1428] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:40:04.925576 8960 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:04.925799 8960 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:40:04.926717 8942 ts_tablet_manager.cc:1397] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:40:04.927389 8942 raft_consensus.cc:348] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.927560 8942 raft_consensus.cc:374] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:04.927670 8942 raft_consensus.cc:729] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d8991f8c67403091da063700f0e381, State: Initialized, Role: FOLLOWER
I20260927 21:40:04.927933 8942 consensus_queue.cc:260] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:04.928323 8947 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:04.928612 8947 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:40:04.928483 8942 ts_tablet_manager.cc:1428] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:40:05.219956 8947 consensus_queue.cc:1035] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:05.226889 8971 consensus_queue.cc:1035] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:05.277192 8946 raft_consensus.cc:482] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:05.277419 8946 raft_consensus.cc:504] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.279284 8946 leader_election.cc:288] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:05.282171 8947 raft_consensus.cc:482] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:05.282387 8947 raft_consensus.cc:504] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.283196 8947 leader_election.cc:288] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:05.283636 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97" is_pre_election: true
I20260927 21:40:05.283901 8729 raft_consensus.cc:2454] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 0.
I20260927 21:40:05.283890 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" is_pre_election: true
I20260927 21:40:05.284181 8868 raft_consensus.cc:2454] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 0.
I20260927 21:40:05.284710 8518 leader_election.cc:302] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:05.285040 8947 raft_consensus.cc:2781] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:05.285171 8947 raft_consensus.cc:482] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:05.285270 8947 raft_consensus.cc:3037] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:05.288194 8988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.279537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:32809 (local address 127.0.202.130:60365)
0927 21:40:05.280778 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:05.280805 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:05.280998 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:05.281509 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:05.281517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:05.281552 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:05.281916 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.281932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.283429 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.283435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.285752 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.285763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.287335 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.287341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.287518 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.287554 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:40:05.287645 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:40:05.287805 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":28,"thread_start_us":891,"threads_started":1}
I20260927 21:40:05.288822 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.279652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:60365 (local address 127.0.202.129:32809)
0927 21:40:05.281121 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:05.281131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:05.281155 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:40:05.281237 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:05.281243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:05.281347 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:05.281458 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:05.282121 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.283287 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.285909 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.287015 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.287174 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.287253 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:05.288370 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0927 21:40:05.288514 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":1156,"threads_started":1}
I20260927 21:40:05.289129 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.279642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41375 (local address 127.0.202.130:56105)
0927 21:40:05.281303 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:05.281322 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:05.281461 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:05.281862 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:05.281866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:05.281891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:05.282390 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.282402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.283949 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.283954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.286411 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.286426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.288146 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.288156 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.288504 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.288547 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:40:05.288635 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:40:05.288758 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":30,"thread_start_us":1349,"threads_started":1}
I20260927 21:40:05.289158 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.279945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:56105 (local address 127.0.202.131:41375)
0927 21:40:05.280231 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:05.280241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:05.281460 (+ 1219us) server_negotiation.cc:407] Connection header received
0927 21:40:05.281598 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:05.281606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:05.281697 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:05.281812 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:05.282600 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.283771 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.286608 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.287806 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.287952 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.288017 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:05.288688 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 21:40:05.288865 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 21:40:05.288959 8947 raft_consensus.cc:504] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.289361 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "17c503e92ef443b991e0e33272258a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381" is_pre_election: true
I20260927 21:40:05.289762 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "17c503e92ef443b991e0e33272258a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" is_pre_election: true
I20260927 21:40:05.289991 8868 raft_consensus.cc:2454] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17c503e92ef443b991e0e33272258a97 in term 0.
I20260927 21:40:05.290518 8585 raft_consensus.cc:2379] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17c503e92ef443b991e0e33272258a97 in current term 1: Already voted for candidate 39d8991f8c67403091da063700f0e381 in this term.
I20260927 21:40:05.290596 8658 leader_election.cc:302] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c503e92ef443b991e0e33272258a97, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:05.291106 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306"
I20260927 21:40:05.291244 8946 raft_consensus.cc:2781] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:05.291337 8868 raft_consensus.cc:3037] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:05.291366 8946 raft_consensus.cc:482] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:05.291468 8946 raft_consensus.cc:3037] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:05.292622 8947 leader_election.cc:288] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 election: Requested vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:05.292596 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97"
I20260927 21:40:05.294623 8868 raft_consensus.cc:2454] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 1.
I20260927 21:40:05.294667 8946 raft_consensus.cc:504] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.295113 8729 raft_consensus.cc:2379] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39d8991f8c67403091da063700f0e381 in current term 1: Already voted for candidate 17c503e92ef443b991e0e33272258a97 in this term.
I20260927 21:40:05.295094 8518 leader_election.cc:302] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:05.295446 8947 raft_consensus.cc:2781] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:05.295841 8947 raft_consensus.cc:686] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [term 1 LEADER]: Becoming Leader. State: Replica: 39d8991f8c67403091da063700f0e381, State: Running, Role: LEADER
I20260927 21:40:05.296125 8946 leader_election.cc:288] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [CANDIDATE]: Term 1 election: Requested vote from peers 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:05.296155 8947 consensus_queue.cc:237] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.296731 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "17c503e92ef443b991e0e33272258a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381"
I20260927 21:40:05.297114 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7411778aab44bf1809fb24affb448b2" candidate_uuid: "17c503e92ef443b991e0e33272258a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306"
I20260927 21:40:05.297418 8868 raft_consensus.cc:2379] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 17c503e92ef443b991e0e33272258a97 in current term 1: Already voted for candidate 39d8991f8c67403091da063700f0e381 in this term.
I20260927 21:40:05.297890 8658 leader_election.cc:302] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17c503e92ef443b991e0e33272258a97; no voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306
I20260927 21:40:05.298161 8946 raft_consensus.cc:2726] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 21:40:05.298763 8971 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: TxnStatusTablet state changed. Reason: New leader 39d8991f8c67403091da063700f0e381. Latest consensus state: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:05.299110 8971 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:40:05.299690 8992 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:40:05.299937 8992 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:40:05.300596 8427 catalog_manager.cc:5495] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39d8991f8c67403091da063700f0e381 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } health_report { overall_health: UNKNOWN } } }
I20260927 21:40:05.390841 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.384531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.0.202.129:32809)
0927 21:40:05.384754 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:05.384759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:05.384777 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:40:05.385089 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:05.385097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:05.385287 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:05.385411 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:05.385792 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.386625 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.388571 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.388999 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.389091 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.389662 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:05.390253 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:05.390373 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:05.390553 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:40:05.390625 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:40:05.390956 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.384461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:32809 (local address 127.0.0.1:43032)
0927 21:40:05.384691 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:05.384713 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:05.384862 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:05.385456 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:05.385462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:05.385475 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:40:05.385664 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.385674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.386763 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.386769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.388433 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.388449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.389146 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.389152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.389379 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.389454 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:40:05.390370 (+ 916us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:40:05.390391 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:40:05.390484 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:40:05.390642 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:40:05.394891 8868 raft_consensus.cc:1261] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Refusing update from remote peer 39d8991f8c67403091da063700f0e381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:05.394914 8729 raft_consensus.cc:1261] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Refusing update from remote peer 39d8991f8c67403091da063700f0e381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:05.395591 8947 consensus_queue.cc:1035] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:05.395866 8971 consensus_queue.cc:1035] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:05.399309 8946 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: TxnStatusTablet state changed. Reason: New leader 39d8991f8c67403091da063700f0e381. Latest consensus state: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:05.399607 8946 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:40:05.399729 8960 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: TxnStatusTablet state changed. Reason: New leader 39d8991f8c67403091da063700f0e381. Latest consensus state: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:05.399991 8960 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:40:05.401888 8984 mvcc.cc:205] Tried to move back new op lower bound from 7334073161291345920 to 7334073160898871296. Current Snapshot: MvccSnapshot[applied={T|T < 7334073161291345920}]
I20260927 21:40:05.404163 8987 mvcc.cc:205] Tried to move back new op lower bound from 7334073161291345920 to 7334073160898871296. Current Snapshot: MvccSnapshot[applied={T|T < 7334073161291345920}]
I20260927 21:40:05.408878 8946 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:05.409166 8946 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 17c503e92ef443b991e0e33272258a97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:40:05.410670 8947 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:05.410934 8947 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:40:05.411739 8960 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:05.412047 8960 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 8ddb4a7a6e574e6bab24f2128ff2f306: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:40:05.414438 8947 tablet_replica.cc:439] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39d8991f8c67403091da063700f0e381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } } }
I20260927 21:40:05.414649 8947 tablet_replica.cc:442] T d7411778aab44bf1809fb24affb448b2 P 39d8991f8c67403091da063700f0e381: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:40:05.436844 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.429387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.202.129:32809)
0927 21:40:05.429761 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:05.429768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:05.430633 (+ 865us) server_negotiation.cc:407] Connection header received
0927 21:40:05.430829 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:05.430838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:05.430940 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:05.431033 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:05.431580 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.432751 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.434793 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.435474 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.435593 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.436061 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:05.436196 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:05.436285 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:05.436452 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 21:40:05.436532 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260927 21:40:05.454980 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.447967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:60115 (local address 127.0.202.129:32809)
0927 21:40:05.448190 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:05.448197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:05.448597 (+ 400us) server_negotiation.cc:407] Connection header received
0927 21:40:05.448811 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:05.448816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:05.448885 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:05.449008 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:05.449601 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.450595 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.452779 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.453507 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.453601 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.454266 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:05.454381 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:05.454470 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:05.454628 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:40:05.454693 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:40:05.455420 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.447901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:32809 (local address 127.0.202.129:60115)
0927 21:40:05.448497 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:05.448512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:05.448685 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:05.449099 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:05.449105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:05.449119 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:40:05.449428 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.449440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.450741 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.450747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.452629 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:05.452644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.453649 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.453656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:05.453831 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.453903 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:40:05.454490 (+ 587us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:40:05.454501 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:40:05.454596 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:40:05.454740 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":321,"threads_started":1}
I20260927 21:40:05.525933 8427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:05.526248 8427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:05.529553 8427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Sending DeleteTablet for 3 replicas of tablet 5680539a19954d3680f4e4e496cfd3ca
I20260927 21:40:05.530617 8703 tablet_service.cc:1514] Processing DeleteTablet for tablet 5680539a19954d3680f4e4e496cfd3ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:05 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:46540
I20260927 21:40:05.530797 8565 tablet_service.cc:1514] Processing DeleteTablet for tablet 5680539a19954d3680f4e4e496cfd3ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:05 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:43022
I20260927 21:40:05.530810 8844 tablet_service.cc:1514] Processing DeleteTablet for tablet 5680539a19954d3680f4e4e496cfd3ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:05 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:35020
I20260927 21:40:05.542037 9010 tablet_replica.cc:330] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: stopping tablet replica
I20260927 21:40:05.542299 9008 tablet_replica.cc:330] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: stopping tablet replica
I20260927 21:40:05.542297 9010 raft_consensus.cc:2229] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:40:05.542626 9008 raft_consensus.cc:2229] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:05.542966 9008 raft_consensus.cc:2258] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:05.542954 9010 raft_consensus.cc:2258] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:05.543627 9009 tablet_replica.cc:330] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: stopping tablet replica
I20260927 21:40:05.544209 9008 ts_tablet_manager.cc:1905] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:05.544538 9009 raft_consensus.cc:2229] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:05.545141 9010 ts_tablet_manager.cc:1905] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:05.545445 9009 raft_consensus.cc:2258] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:05.550354 9009 ts_tablet_manager.cc:1905] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:05.552006 9008 ts_tablet_manager.cc:1918] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:40:05.552187 9008 log.cc:1196] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/5680539a19954d3680f4e4e496cfd3ca
I20260927 21:40:05.552865 9008 ts_tablet_manager.cc:1939] T 5680539a19954d3680f4e4e496cfd3ca P 17c503e92ef443b991e0e33272258a97: Deleting consensus metadata
I20260927 21:40:05.554319 8413 catalog_manager.cc:4841] TS 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887): tablet 5680539a19954d3680f4e4e496cfd3ca (table test-table [id=1c91bc4e7a9540878f70c505bcfcdb0e]) successfully deleted
I20260927 21:40:05.558976 9009 ts_tablet_manager.cc:1918] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:40:05.559149 9009 log.cc:1196] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/5680539a19954d3680f4e4e496cfd3ca
I20260927 21:40:05.560537 9010 ts_tablet_manager.cc:1918] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:40:05.560750 9010 log.cc:1196] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/5680539a19954d3680f4e4e496cfd3ca
I20260927 21:40:05.560832 9009 ts_tablet_manager.cc:1939] T 5680539a19954d3680f4e4e496cfd3ca P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting consensus metadata
I20260927 21:40:05.561278 9010 ts_tablet_manager.cc:1939] T 5680539a19954d3680f4e4e496cfd3ca P 39d8991f8c67403091da063700f0e381: Deleting consensus metadata
I20260927 21:40:05.562084 8415 catalog_manager.cc:4841] TS 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375): tablet 5680539a19954d3680f4e4e496cfd3ca (table test-table [id=1c91bc4e7a9540878f70c505bcfcdb0e]) successfully deleted
I20260927 21:40:05.563112 8412 catalog_manager.cc:4841] TS 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809): tablet 5680539a19954d3680f4e4e496cfd3ca (table test-table [id=1c91bc4e7a9540878f70c505bcfcdb0e]) successfully deleted
Sep 27 21:40:05 dist-test-slave-25dq krb5kdc[8383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:40:05 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:05 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:40:05.603621 9018 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:05.590666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.0.202.190:44379)
0927 21:40:05.591599 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:05.591604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:05.591621 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:40:05.591674 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:05.591677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:05.591735 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:05.591829 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:05.593731 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.594607 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.596448 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.596858 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.596958 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.603309 (+ 6351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49306: BlockingRecv error: recv got EOF from 127.0.0.1:49306 (error 108)
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":666,"threads_started":1}
Sep 27 21:40:05 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM
I20260927 21:40:05.636512 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:05.620069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.0.202.190:44379)
0927 21:40:05.620275 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:05.620282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:05.620728 (+ 446us) server_negotiation.cc:407] Connection header received
0927 21:40:05.620907 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:05.620911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:05.620976 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:05.621063 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:05.622750 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.623587 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.625665 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:05.626139 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:05.626255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:05.631209 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:05.631214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:05.631226 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:05.631264 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:05.633923 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:05.634565 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:05.634576 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:05.634578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:05.634705 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:05.635083 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:05.635098 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:05.635100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:05.635357 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:05.635618 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:05.635917 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 21:40:05.636152 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:40:05.640517 8427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:40:05.641335 8427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:40:05.649268 8565 tablet_service.cc:1467] Processing CreateTablet for tablet 89284f318c0e415e90babd5409893598 (DEFAULT_TABLE table=test-table [id=5ad374d0a97a4a3c8a409fefc243fa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:05.649785 8565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89284f318c0e415e90babd5409893598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:05.652541 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 89284f318c0e415e90babd5409893598 (DEFAULT_TABLE table=test-table [id=5ad374d0a97a4a3c8a409fefc243fa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:05.652580 8844 tablet_service.cc:1467] Processing CreateTablet for tablet 89284f318c0e415e90babd5409893598 (DEFAULT_TABLE table=test-table [id=5ad374d0a97a4a3c8a409fefc243fa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:05.652987 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89284f318c0e415e90babd5409893598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:05.653007 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89284f318c0e415e90babd5409893598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:05.665827 9026 tablet_bootstrap.cc:492] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: Bootstrap starting.
I20260927 21:40:05.668772 9026 tablet_bootstrap.cc:654] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:05.667738 9025 tablet_bootstrap.cc:492] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: Bootstrap starting.
I20260927 21:40:05.670296 9024 tablet_bootstrap.cc:492] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: Bootstrap starting.
I20260927 21:40:05.671605 9025 tablet_bootstrap.cc:654] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:05.672509 9024 tablet_bootstrap.cc:654] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:05.676663 9024 tablet_bootstrap.cc:492] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: No bootstrap required, opened a new log
I20260927 21:40:05.676874 9024 ts_tablet_manager.cc:1397] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260927 21:40:05.678097 9024 raft_consensus.cc:348] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.678467 9024 raft_consensus.cc:374] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:05.681710 9026 tablet_bootstrap.cc:492] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: No bootstrap required, opened a new log
I20260927 21:40:05.681910 9026 ts_tablet_manager.cc:1397] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260927 21:40:05.682931 9026 raft_consensus.cc:348] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.683254 9026 raft_consensus.cc:374] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:05.683377 9026 raft_consensus.cc:729] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c503e92ef443b991e0e33272258a97, State: Initialized, Role: FOLLOWER
I20260927 21:40:05.683717 9026 consensus_queue.cc:260] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.684628 9026 ts_tablet_manager.cc:1428] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:05.685302 9024 raft_consensus.cc:729] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d8991f8c67403091da063700f0e381, State: Initialized, Role: FOLLOWER
I20260927 21:40:05.685832 9024 consensus_queue.cc:260] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.691655 9024 ts_tablet_manager.cc:1428] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20260927 21:40:05.694093 9025 tablet_bootstrap.cc:492] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: No bootstrap required, opened a new log
I20260927 21:40:05.694274 9025 ts_tablet_manager.cc:1397] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.008s
I20260927 21:40:05.694905 9025 raft_consensus.cc:348] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.695124 9025 raft_consensus.cc:374] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:05.695214 9025 raft_consensus.cc:729] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Initialized, Role: FOLLOWER
I20260927 21:40:05.695472 9025 consensus_queue.cc:260] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:05.695955 9025 ts_tablet_manager.cc:1428] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 21:40:06.087474 8971 raft_consensus.cc:482] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:06.087728 8971 raft_consensus.cc:504] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.088537 8971 leader_election.cc:288] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:06.088891 8960 raft_consensus.cc:482] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:06.089076 8960 raft_consensus.cc:504] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.089131 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97" is_pre_election: true
I20260927 21:40:06.089316 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" is_pre_election: true
I20260927 21:40:06.089385 8729 raft_consensus.cc:2454] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 0.
I20260927 21:40:06.089512 8868 raft_consensus.cc:2454] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d8991f8c67403091da063700f0e381 in term 0.
I20260927 21:40:06.089740 8960 leader_election.cc:288] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809), 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887)
I20260927 21:40:06.090049 8518 leader_election.cc:302] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:06.090245 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97" is_pre_election: true
I20260927 21:40:06.090430 8971 raft_consensus.cc:2781] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:06.090445 8729 raft_consensus.cc:2454] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 0.
I20260927 21:40:06.090572 8971 raft_consensus.cc:482] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:06.090678 8971 raft_consensus.cc:3037] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:06.090848 8795 leader_election.cc:302] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c503e92ef443b991e0e33272258a97, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:06.090835 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381" is_pre_election: true
I20260927 21:40:06.091135 8960 raft_consensus.cc:2781] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:06.091243 8960 raft_consensus.cc:482] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:06.091329 8960 raft_consensus.cc:3037] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:06.093385 8971 raft_consensus.cc:504] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.093812 8585 raft_consensus.cc:2379] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in current term 1: Already voted for candidate 39d8991f8c67403091da063700f0e381 in this term.
I20260927 21:40:06.094343 8960 raft_consensus.cc:504] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.094682 8868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306"
I20260927 21:40:06.094954 8868 raft_consensus.cc:2379] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39d8991f8c67403091da063700f0e381 in current term 1: Already voted for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in this term.
I20260927 21:40:06.095394 8971 leader_election.cc:288] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 election: Requested vote from peers 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887), 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375)
I20260927 21:40:06.095578 8960 leader_election.cc:288] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 election: Requested vote from peers 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809), 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887)
I20260927 21:40:06.095541 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97"
I20260927 21:40:06.095597 8728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "39d8991f8c67403091da063700f0e381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97"
I20260927 21:40:06.095736 8729 raft_consensus.cc:3037] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:06.095395 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89284f318c0e415e90babd5409893598" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381"
I20260927 21:40:06.096311 8585 raft_consensus.cc:2379] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in current term 1: Already voted for candidate 39d8991f8c67403091da063700f0e381 in this term.
I20260927 21:40:06.096227 8516 leader_election.cc:302] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39d8991f8c67403091da063700f0e381; no voters: 17c503e92ef443b991e0e33272258a97, 8ddb4a7a6e574e6bab24f2128ff2f306
I20260927 21:40:06.096678 8971 raft_consensus.cc:2726] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 21:40:06.098929 8729 raft_consensus.cc:2454] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 1.
I20260927 21:40:06.099316 8795 leader_election.cc:302] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17c503e92ef443b991e0e33272258a97, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters: 39d8991f8c67403091da063700f0e381
I20260927 21:40:06.099623 8960 raft_consensus.cc:2781] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:06.100067 8960 raft_consensus.cc:686] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 LEADER]: Becoming Leader. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Running, Role: LEADER
I20260927 21:40:06.100430 8960 consensus_queue.cc:237] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.103273 8428 catalog_manager.cc:5495] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } health_report { overall_health: HEALTHY } } }
I20260927 21:40:06.125623 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.118629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.0.202.131:41375)
0927 21:40:06.119181 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:06.119186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:06.119204 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:40:06.119268 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:06.119271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:06.119350 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:06.119440 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:06.119961 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.120971 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.123364 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.123847 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.123952 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.124680 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:06.124807 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:06.124908 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:06.125151 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 21:40:06.125246 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":300,"threads_started":1}
I20260927 21:40:06.129002 8585 raft_consensus.cc:1261] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb4a7a6e574e6bab24f2128ff2f306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:06.129009 8729 raft_consensus.cc:1261] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb4a7a6e574e6bab24f2128ff2f306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:06.129627 9030 consensus_queue.cc:1035] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:06.129952 8960 consensus_queue.cc:1035] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:06.162182 8428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:06.162395 8428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:06.165536 8428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Sending DeleteTablet for 3 replicas of tablet 89284f318c0e415e90babd5409893598
I20260927 21:40:06.166450 8844 tablet_service.cc:1514] Processing DeleteTablet for tablet 89284f318c0e415e90babd5409893598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:06 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:35020
I20260927 21:40:06.166453 8565 tablet_service.cc:1514] Processing DeleteTablet for tablet 89284f318c0e415e90babd5409893598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:06 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:43022
I20260927 21:40:06.166883 8703 tablet_service.cc:1514] Processing DeleteTablet for tablet 89284f318c0e415e90babd5409893598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:06 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:46540
I20260927 21:40:06.167724 9045 tablet_replica.cc:330] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: stopping tablet replica
I20260927 21:40:06.167959 9045 raft_consensus.cc:2229] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:06.167846 8428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:40:06.168272 9045 raft_consensus.cc:2258] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:06.169354 9045 ts_tablet_manager.cc:1905] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:40:06.169607 8428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:40:06.175648 9045 ts_tablet_manager.cc:1918] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:06.175904 9045 log.cc:1196] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/89284f318c0e415e90babd5409893598
I20260927 21:40:06.176450 9045 ts_tablet_manager.cc:1939] T 89284f318c0e415e90babd5409893598 P 17c503e92ef443b991e0e33272258a97: Deleting consensus metadata
I20260927 21:40:06.178812 8564 tablet_service.cc:1467] Processing CreateTablet for tablet 977ad2a15ab74fb6bce5e413986f3905 (DEFAULT_TABLE table=test-table [id=aaf40d5c0e26415a9cc1c544e60d5570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:06.179085 8843 tablet_service.cc:1467] Processing CreateTablet for tablet 977ad2a15ab74fb6bce5e413986f3905 (DEFAULT_TABLE table=test-table [id=aaf40d5c0e26415a9cc1c544e60d5570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:06.179394 8564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977ad2a15ab74fb6bce5e413986f3905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:06.179610 8843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977ad2a15ab74fb6bce5e413986f3905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:06.181185 9044 tablet_replica.cc:330] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: stopping tablet replica
I20260927 21:40:06.181404 9044 raft_consensus.cc:2229] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:40:06.181919 9044 raft_consensus.cc:2258] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:06.182539 9043 tablet_replica.cc:330] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: stopping tablet replica
I20260927 21:40:06.182662 8413 catalog_manager.cc:4841] TS 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887): tablet 89284f318c0e415e90babd5409893598 (table test-table [id=5ad374d0a97a4a3c8a409fefc243fa8d]) successfully deleted
I20260927 21:40:06.182875 9043 raft_consensus.cc:2229] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:06.183188 9043 raft_consensus.cc:2258] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:06.183423 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 977ad2a15ab74fb6bce5e413986f3905 (DEFAULT_TABLE table=test-table [id=aaf40d5c0e26415a9cc1c544e60d5570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:06.183852 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977ad2a15ab74fb6bce5e413986f3905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:06.185251 9043 ts_tablet_manager.cc:1905] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:06.187544 9024 tablet_bootstrap.cc:492] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: Bootstrap starting.
I20260927 21:40:06.188481 9044 ts_tablet_manager.cc:1905] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:06.188616 9025 tablet_bootstrap.cc:492] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: Bootstrap starting.
I20260927 21:40:06.189635 9024 tablet_bootstrap.cc:654] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:06.190402 9025 tablet_bootstrap.cc:654] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:06.196069 9043 ts_tablet_manager.cc:1918] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:06.196069 9044 ts_tablet_manager.cc:1918] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:06.196231 9044 log.cc:1196] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/89284f318c0e415e90babd5409893598
I20260927 21:40:06.196317 9043 log.cc:1196] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/89284f318c0e415e90babd5409893598
I20260927 21:40:06.196902 9044 ts_tablet_manager.cc:1939] T 89284f318c0e415e90babd5409893598 P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting consensus metadata
I20260927 21:40:06.198107 9043 ts_tablet_manager.cc:1939] T 89284f318c0e415e90babd5409893598 P 39d8991f8c67403091da063700f0e381: Deleting consensus metadata
I20260927 21:40:06.198180 8415 catalog_manager.cc:4841] TS 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375): tablet 89284f318c0e415e90babd5409893598 (table test-table [id=5ad374d0a97a4a3c8a409fefc243fa8d]) successfully deleted
I20260927 21:40:06.199229 8412 catalog_manager.cc:4841] TS 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809): tablet 89284f318c0e415e90babd5409893598 (table test-table [id=5ad374d0a97a4a3c8a409fefc243fa8d]) successfully deleted
I20260927 21:40:06.199785 9025 tablet_bootstrap.cc:492] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: No bootstrap required, opened a new log
I20260927 21:40:06.200012 9025 ts_tablet_manager.cc:1397] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20260927 21:40:06.201128 9025 raft_consensus.cc:348] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.201525 9025 raft_consensus.cc:374] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:06.201637 9025 raft_consensus.cc:729] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Initialized, Role: FOLLOWER
I20260927 21:40:06.201957 9025 consensus_queue.cc:260] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.202732 9025 ts_tablet_manager.cc:1428] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260927 21:40:06.204953 9048 tablet_bootstrap.cc:492] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: Bootstrap starting.
I20260927 21:40:06.206473 9024 tablet_bootstrap.cc:492] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: No bootstrap required, opened a new log
I20260927 21:40:06.206610 9024 ts_tablet_manager.cc:1397] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.003s
I20260927 21:40:06.206880 9048 tablet_bootstrap.cc:654] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:06.207163 9024 raft_consensus.cc:348] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.207350 9024 raft_consensus.cc:374] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:06.207396 9024 raft_consensus.cc:729] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d8991f8c67403091da063700f0e381, State: Initialized, Role: FOLLOWER
I20260927 21:40:06.207633 9024 consensus_queue.cc:260] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.208357 9024 ts_tablet_manager.cc:1428] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 21:40:06.210302 9048 tablet_bootstrap.cc:492] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: No bootstrap required, opened a new log
I20260927 21:40:06.210485 9048 ts_tablet_manager.cc:1397] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 21:40:06.211138 9048 raft_consensus.cc:348] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.211344 9048 raft_consensus.cc:374] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:06.211462 9048 raft_consensus.cc:729] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c503e92ef443b991e0e33272258a97, State: Initialized, Role: FOLLOWER
I20260927 21:40:06.211717 9048 consensus_queue.cc:260] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.212234 9048 ts_tablet_manager.cc:1428] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:40:06.372699 9042 raft_consensus.cc:482] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:06.372997 9042 raft_consensus.cc:504] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.373891 9042 leader_election.cc:288] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809), 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887)
I20260927 21:40:06.374648 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977ad2a15ab74fb6bce5e413986f3905" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97" is_pre_election: true
I20260927 21:40:06.374737 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977ad2a15ab74fb6bce5e413986f3905" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381" is_pre_election: true
I20260927 21:40:06.374975 8585 raft_consensus.cc:2454] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 0.
I20260927 21:40:06.374959 8729 raft_consensus.cc:2454] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 0.
I20260927 21:40:06.375466 8794 leader_election.cc:302] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:06.375818 9042 raft_consensus.cc:2781] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:06.375916 9042 raft_consensus.cc:482] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:06.376029 9042 raft_consensus.cc:3037] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:06.378662 9042 raft_consensus.cc:504] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.379303 9042 leader_election.cc:288] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 election: Requested vote from peers 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809), 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887)
I20260927 21:40:06.379943 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977ad2a15ab74fb6bce5e413986f3905" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d8991f8c67403091da063700f0e381"
I20260927 21:40:06.380173 8585 raft_consensus.cc:3037] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:06.380038 8729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977ad2a15ab74fb6bce5e413986f3905" candidate_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c503e92ef443b991e0e33272258a97"
I20260927 21:40:06.380355 8729 raft_consensus.cc:3037] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:06.382246 8585 raft_consensus.cc:2454] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 1.
I20260927 21:40:06.382344 8729 raft_consensus.cc:2454] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddb4a7a6e574e6bab24f2128ff2f306 in term 1.
I20260927 21:40:06.382696 8794 leader_election.cc:302] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d8991f8c67403091da063700f0e381, 8ddb4a7a6e574e6bab24f2128ff2f306; no voters:
I20260927 21:40:06.383005 9042 raft_consensus.cc:2781] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:06.383193 9042 raft_consensus.cc:686] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 LEADER]: Becoming Leader. State: Replica: 8ddb4a7a6e574e6bab24f2128ff2f306, State: Running, Role: LEADER
I20260927 21:40:06.383476 9042 consensus_queue.cc:237] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } }
I20260927 21:40:06.386644 8428 catalog_manager.cc:5495] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddb4a7a6e574e6bab24f2128ff2f306" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41375 } health_report { overall_health: HEALTHY } } }
I20260927 21:40:06.454150 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.447752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:32809 (local address 127.0.202.131:44619)
0927 21:40:06.448437 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:06.448485 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:06.448653 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:06.449208 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:06.449212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:06.449222 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:40:06.449500 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:06.449507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.450642 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.450652 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:06.452002 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:06.452014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.452907 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.452914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:06.453047 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.453108 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:40:06.453529 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:40:06.453546 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 21:40:06.453641 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:40:06.453779 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":383,"threads_started":1}
I20260927 21:40:06.454154 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.448060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:44619 (local address 127.0.202.129:32809)
0927 21:40:06.448873 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:06.448878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:06.448898 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:40:06.448964 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:06.448968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:06.449028 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:06.449118 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:06.449682 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.450491 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.452154 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.452713 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.452871 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.453307 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:06.453408 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:06.453504 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:06.453662 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:40:06.453737 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":548,"threads_started":1}
I20260927 21:40:06.463800 8729 raft_consensus.cc:1261] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb4a7a6e574e6bab24f2128ff2f306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:06.463820 8585 raft_consensus.cc:1261] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Refusing update from remote peer 8ddb4a7a6e574e6bab24f2128ff2f306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:06.464450 9042 consensus_queue.cc:1035] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d8991f8c67403091da063700f0e381" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:06.464726 9030 consensus_queue.cc:1035] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c503e92ef443b991e0e33272258a97" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:06.494244 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.487546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:53827 (local address 127.0.202.131:41375)
0927 21:40:06.487726 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:06.487735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:06.488419 (+ 684us) server_negotiation.cc:407] Connection header received
0927 21:40:06.488611 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:06.488617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:06.488710 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:06.488844 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:06.489423 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.490153 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.491993 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.492801 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.492959 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.493409 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:06.493608 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:06.493715 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:06.493856 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:40:06.493942 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:40:06.494191 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.487584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41375 (local address 127.0.202.129:53827)
0927 21:40:06.488270 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:06.488318 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:06.488471 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:06.488896 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:06.488902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:06.488917 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:40:06.489213 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:06.489226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.490362 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.490374 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:06.491848 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:06.491863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.492994 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.492998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:06.493125 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.493163 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:40:06.493723 (+ 560us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:40:06.493734 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:40:06.493800 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:40:06.493908 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":392,"threads_started":1}
I20260927 21:40:06.524760 8428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:06.525043 8428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:06.528116 8428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df33ef9edcfc4d169c15a316ae2dc261: Sending DeleteTablet for 3 replicas of tablet 977ad2a15ab74fb6bce5e413986f3905
I20260927 21:40:06.529044 8564 tablet_service.cc:1514] Processing DeleteTablet for tablet 977ad2a15ab74fb6bce5e413986f3905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:06 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:43022
I20260927 21:40:06.529157 8843 tablet_service.cc:1514] Processing DeleteTablet for tablet 977ad2a15ab74fb6bce5e413986f3905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:06 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:35020
I20260927 21:40:06.529393 8703 tablet_service.cc:1514] Processing DeleteTablet for tablet 977ad2a15ab74fb6bce5e413986f3905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:06 UTC) from {username='oryx', principal='oryx/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:46540
I20260927 21:40:06.529554 9044 tablet_replica.cc:330] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: stopping tablet replica
I20260927 21:40:06.529698 9045 tablet_replica.cc:330] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: stopping tablet replica
I20260927 21:40:06.529768 9044 raft_consensus.cc:2229] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:40:06.529951 9045 raft_consensus.cc:2229] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:06.530119 9044 raft_consensus.cc:2258] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:06.530179 9045 raft_consensus.cc:2258] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:06.530675 9043 tablet_replica.cc:330] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: stopping tablet replica
I20260927 21:40:06.530930 9043 raft_consensus.cc:2229] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:06.531116 9043 raft_consensus.cc:2258] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:06.531417 9044 ts_tablet_manager.cc:1905] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:06.532092 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 8499
I20260927 21:40:06.532403 9045 ts_tablet_manager.cc:1905] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:06.532573 9043 ts_tablet_manager.cc:1905] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:06.538733 9043 ts_tablet_manager.cc:1918] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:40:06.538914 9043 log.cc:1196] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/977ad2a15ab74fb6bce5e413986f3905
I20260927 21:40:06.539408 9043 ts_tablet_manager.cc:1939] T 977ad2a15ab74fb6bce5e413986f3905 P 39d8991f8c67403091da063700f0e381: Deleting consensus metadata
I20260927 21:40:06.539543 9045 ts_tablet_manager.cc:1918] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:40:06.539543 9044 ts_tablet_manager.cc:1918] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:40:06.539706 9044 log.cc:1196] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/977ad2a15ab74fb6bce5e413986f3905
I20260927 21:40:06.539804 9045 log.cc:1196] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/977ad2a15ab74fb6bce5e413986f3905
I20260927 21:40:06.540335 9044 ts_tablet_manager.cc:1939] T 977ad2a15ab74fb6bce5e413986f3905 P 8ddb4a7a6e574e6bab24f2128ff2f306: Deleting consensus metadata
I20260927 21:40:06.540349 9045 ts_tablet_manager.cc:1939] T 977ad2a15ab74fb6bce5e413986f3905 P 17c503e92ef443b991e0e33272258a97: Deleting consensus metadata
I20260927 21:40:06.540629 8412 catalog_manager.cc:4841] TS 39d8991f8c67403091da063700f0e381 (127.0.202.129:32809): tablet 977ad2a15ab74fb6bce5e413986f3905 (table test-table [id=aaf40d5c0e26415a9cc1c544e60d5570]) successfully deleted
I20260927 21:40:06.541581 8415 catalog_manager.cc:4841] TS 8ddb4a7a6e574e6bab24f2128ff2f306 (127.0.202.131:41375): tablet 977ad2a15ab74fb6bce5e413986f3905 (table test-table [id=aaf40d5c0e26415a9cc1c544e60d5570]) successfully deleted
I20260927 21:40:06.541632 8413 catalog_manager.cc:4841] TS 17c503e92ef443b991e0e33272258a97 (127.0.202.130:40887): tablet 977ad2a15ab74fb6bce5e413986f3905 (table test-table [id=aaf40d5c0e26415a9cc1c544e60d5570]) successfully deleted
Sep 27 21:40:06 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.129@KRBTEST.COM
I20260927 21:40:06.553393 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.536849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.0.202.129:32809)
0927 21:40:06.537114 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:06.537122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:06.537361 (+ 239us) server_negotiation.cc:407] Connection header received
0927 21:40:06.537650 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:06.537657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:06.537736 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:06.537807 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:06.539895 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.541120 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.542669 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.543088 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.543211 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.547278 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:06.547302 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:06.547318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:06.547355 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:06.550463 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:06.551104 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:06.551112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:06.551114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:06.551245 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:06.551622 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:06.551626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:06.551628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:06.552107 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:06.552522 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:06.552858 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 21:40:06.553053 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 21:40:06.554243 8630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:06.707304 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 8499
I20260927 21:40:06.736799 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 8639
Sep 27 21:40:06 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.130@KRBTEST.COM
I20260927 21:40:06.754463 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.737845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.0.202.130:40887)
0927 21:40:06.738369 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:06.738376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:06.738393 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:40:06.738445 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:06.738449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:06.738506 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:06.738606 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:06.739920 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.740875 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.742472 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.742863 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.742961 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.747500 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:06.747515 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:06.747532 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:06.747573 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:06.751076 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:06.751817 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:06.751825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:06.751827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:06.752049 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:06.752681 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:06.752689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:06.752696 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:06.753317 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:06.753559 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:06.753919 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 21:40:06.754129 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":285,"threads_started":1}
I20260927 21:40:06.755146 8769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:06.856129 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 8639
I20260927 21:40:06.875962 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 8778
Sep 27 21:40:06 dist-test-slave-25dq krb5kdc[8383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.202.131@KRBTEST.COM
I20260927 21:40:06.892551 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:06.876962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.0.202.131:41375)
0927 21:40:06.877159 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:06.877166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:06.877189 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:40:06.877253 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:06.877257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:06.877321 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:06.877414 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:06.878791 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.879557 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.881167 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:06.881573 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:06.881676 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:06.886168 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:06.886185 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:06.886202 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:06.886238 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:06.889311 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:06.890319 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:06.890328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:06.890330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:06.890456 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:06.890872 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:06.890876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:06.890878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:06.891347 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:06.891597 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:06.891921 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:40:06.892105 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:40:06.893235 8909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:07.017215 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 8778
I20260927 21:40:07.039546 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 8396
I20260927 21:40:07.040827 8468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:07.155580 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 8396
2026-09-27T21:40:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7798 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:40:07 dist-test-slave-25dq krb5kdc[9071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:40:07 dist-test-slave-25dq krb5kdc[9071](info): set up 2 sockets
Sep 27 21:40:07 dist-test-slave-25dq krb5kdc[9071](info): commencing operation
krb5kdc: starting...
W20260927 21:40:08.294083 810 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:40:08 dist-test-slave-25dq krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:40:08.541132 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41757
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:41757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:08.871163 9084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:08.871596 9084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:08.871757 9084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:08.879156 9084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:40:08.879333 9084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:08.879503 9084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:08.879616 9084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:40:08.879710 9084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:40:08.889985 9084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:41757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:08.891430 9084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:08.892849 9084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:08.905730 9089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:08.905653 9092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:08.905686 9090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:08.906073 9084 server_base.cc:1034] running on GCE node
I20260927 21:40:08.907101 9084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:08.908654 9084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:08.909883 9084 hybrid_clock.cc:648] HybridClock initialized: now 1790545208909808 us; error 58 us; skew 500 ppm
I20260927 21:40:08.910283 9084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T21:40:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1823 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:40:09 dist-test-slave-25dq krb5kdc[9098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:40:09 dist-test-slave-25dq krb5kdc[9098](info): set up 2 sockets
Sep 27 21:40:09 dist-test-slave-25dq krb5kdc[9098](info): commencing operation
krb5kdc: starting...
W20260927 21:40:10.169781 810 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:40:10 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:40:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:40:10.410192 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:37521
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:37521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:10.750900 9111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:10.751257 9111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:10.751435 9111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:10.758785 9111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:40:10.758963 9111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:10.759092 9111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:10.759189 9111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:40:10.759277 9111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:40:10.769544 9111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:37521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:37521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:10.771139 9111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:10.772732 9111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:10.786067 9116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:10.785954 9119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:10.786015 9117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:10.787364 9111 server_base.cc:1034] running on GCE node
I20260927 21:40:10.788210 9111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:10.789899 9111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:10.791246 9111 hybrid_clock.cc:648] HybridClock initialized: now 1790545210791041 us; error 107 us; skew 500 ppm
I20260927 21:40:10.791648 9111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:10 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:10.800427 9111 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:40:10.802138 9111 webserver.cc:460] Webserver started at http://127.0.202.190:43031/ using document root <none> and password file <none>
I20260927 21:40:10.802954 9111 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:10.803112 9111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:10.803457 9111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:10.806816 9111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "c4711d71f16c4891a6e6b814249fba15"
format_stamp: "Formatted at 2026-09-27 21:40:10 on dist-test-slave-25dq"
server_key: "6ae9ed4687a2e8f0a1282c2eda8b6270"
server_key_iv: "d12570caa674784904db38902c9d281e"
server_key_version: "encryptionkey@0"
I20260927 21:40:10.807704 9111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "c4711d71f16c4891a6e6b814249fba15"
format_stamp: "Formatted at 2026-09-27 21:40:10 on dist-test-slave-25dq"
server_key: "6ae9ed4687a2e8f0a1282c2eda8b6270"
server_key_iv: "d12570caa674784904db38902c9d281e"
server_key_version: "encryptionkey@0"
I20260927 21:40:10.813684 9111 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 21:40:10.817747 9126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:10.819379 9111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:40:10.819605 9111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "c4711d71f16c4891a6e6b814249fba15"
format_stamp: "Formatted at 2026-09-27 21:40:10 on dist-test-slave-25dq"
server_key: "6ae9ed4687a2e8f0a1282c2eda8b6270"
server_key_iv: "d12570caa674784904db38902c9d281e"
server_key_version: "encryptionkey@0"
I20260927 21:40:10.819840 9111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:10.837946 9111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:10.842732 9111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:10.843042 9111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:10.866135 9111 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:37521
I20260927 21:40:10.868270 9111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:40:10.866103 9177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:37521 every 8 connection(s)
I20260927 21:40:10.873813 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9111
I20260927 21:40:10.874044 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:40:10.874397 9178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:10.874826 810 external_mini_cluster.cc:1428] Setting key 40c3c76cad88c2da8b020604f0a1485a
I20260927 21:40:10.893134 9178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Bootstrap starting.
Sep 27 21:40:10 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:10.898265 9178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:10.899881 9178 log.cc:824] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:10.903733 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:10.878055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.0.202.190:37521)
0927 21:40:10.878945 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:10.878959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:10.879005 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:40:10.880935 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:10.880962 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:10.881411 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:10.881956 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:10.883860 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:10.886090 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:10.887606 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:10.888482 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:10.888664 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:10.894621 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:10.894631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:10.894648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:10.894691 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:10.898753 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:10.899625 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:10.899634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:10.899644 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:10.899829 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:10.900252 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:10.900258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:10.900261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:10.901138 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:10.901462 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:10.902038 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 21:40:10.902596 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":408,"threads_started":1}
I20260927 21:40:10.904811 9178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: No bootstrap required, opened a new log
I20260927 21:40:10.913329 9178 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:10.913838 9178 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:10.913964 9178 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4711d71f16c4891a6e6b814249fba15, State: Initialized, Role: FOLLOWER
I20260927 21:40:10.914600 9178 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:10.914852 9178 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:40:10.914988 9178 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:40:10.915158 9178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:10.917567 9178 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:10.918227 9178 leader_election.cc:302] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4711d71f16c4891a6e6b814249fba15; no voters:
I20260927 21:40:10.918838 9178 leader_election.cc:288] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:40:10.919018 9183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:10.919719 9183 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 LEADER]: Becoming Leader. State: Replica: c4711d71f16c4891a6e6b814249fba15, State: Running, Role: LEADER
I20260927 21:40:10.920275 9183 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:10.920989 9178 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:40:10.923466 9184 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4711d71f16c4891a6e6b814249fba15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } } }
I20260927 21:40:10.924041 9184 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:10.923889 9185 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4711d71f16c4891a6e6b814249fba15. Latest consensus state: current_term: 1 leader_uuid: "c4711d71f16c4891a6e6b814249fba15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } } }
I20260927 21:40:10.924542 9185 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:10.927030 9192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:40:10.932587 9192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:40:10.942484 9192 catalog_manager.cc:1347] Generated new cluster ID: 4b27689abd9940b98d1fb63b2aac3adf
I20260927 21:40:10.942590 9192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:40:10.984491 9192 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:40:10.986037 9192 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:40:11.002087 9192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Generated new TSK 0
I20260927 21:40:11.003139 9192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:40:11.076020 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:37521
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:11.414731 9204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:11.415020 9204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:11.415172 9204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:11.422698 9204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:11.422802 9204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:11.422969 9204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:40:11.434011 9204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:11.435335 9204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:11.436815 9204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:11.449851 9212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:11.449946 9209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:11.449930 9210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:11.450628 9204 server_base.cc:1034] running on GCE node
I20260927 21:40:11.451200 9204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:11.452158 9204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:11.453402 9204 hybrid_clock.cc:648] HybridClock initialized: now 1790545211453332 us; error 73 us; skew 500 ppm
I20260927 21:40:11.453725 9204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:11 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:11.457279 9204 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:40:11.458920 9204 webserver.cc:460] Webserver started at http://127.0.202.129:41423/ using document root <none> and password file <none>
I20260927 21:40:11.459630 9204 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:11.459734 9204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:11.460006 9204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:11.463157 9204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "1668b497740a4b88a9db3efe74a890bb"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "007c6a478c332a7f1e064a38fb91ae6e"
server_key_iv: "1f25b7ce425ce98047db2039fa4bc341"
server_key_version: "encryptionkey@0"
I20260927 21:40:11.463935 9204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "1668b497740a4b88a9db3efe74a890bb"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "007c6a478c332a7f1e064a38fb91ae6e"
server_key_iv: "1f25b7ce425ce98047db2039fa4bc341"
server_key_version: "encryptionkey@0"
I20260927 21:40:11.469487 9204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:40:11.473213 9219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:11.474686 9204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:40:11.474911 9204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "1668b497740a4b88a9db3efe74a890bb"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "007c6a478c332a7f1e064a38fb91ae6e"
server_key_iv: "1f25b7ce425ce98047db2039fa4bc341"
server_key_version: "encryptionkey@0"
I20260927 21:40:11.475102 9204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:11.486059 9204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:11.490617 9204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:11.490943 9204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:11.491981 9204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:11.493398 9204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:11.493471 9204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:11.493587 9204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:11.493654 9204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:11.530030 9204 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41457
I20260927 21:40:11.530153 9331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41457 every 8 connection(s)
I20260927 21:40:11.531375 9204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:40:11.538331 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9204
I20260927 21:40:11.538537 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:40:11.539229 810 external_mini_cluster.cc:1428] Setting key 2a56406da6190055342c6012d1bb8444
Sep 27 21:40:11 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:11.553866 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:11.534134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:55209 (local address 127.0.202.190:37521)
0927 21:40:11.534723 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:11.534727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:11.535508 (+ 781us) server_negotiation.cc:407] Connection header received
0927 21:40:11.537698 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:11.537704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:11.537801 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:11.537965 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:11.539723 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:11.540735 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:11.542650 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:11.543421 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:11.543613 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:11.547220 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:11.547229 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:11.547237 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:11.547284 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:11.550296 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:11.551303 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:11.551308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:11.551310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:11.551442 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:11.551998 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:11.552002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:11.552004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:11.552278 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:11.552478 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:11.553386 (+ 908us) server_negotiation.cc:299] Negotiation successful
0927 21:40:11.553586 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":390,"threads_started":1}
I20260927 21:40:11.555440 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:11.534620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:55209)
0927 21:40:11.535342 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:11.535382 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:11.537394 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:11.538201 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:11.538209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:11.538903 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:11.539535 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:11.539554 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:11.540901 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:11.540914 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:11.542486 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:11.542499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:11.543664 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:11.543669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:11.543881 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:11.544701 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:11.544724 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:11.546969 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:11.550455 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:11.550467 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:11.550489 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:11.551173 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:11.551563 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:11.551570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:11.551577 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:11.551838 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:11.552479 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:11.552490 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:11.552984 (+ 494us) client_negotiation.cc:769] Sending connection context
0927 21:40:11.553303 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 21:40:11.553910 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":268,"threads_started":1}
I20260927 21:40:11.557265 9332 heartbeater.cc:347] Connected to a master server at 127.0.202.190:37521
I20260927 21:40:11.557657 9332 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:11.558395 9332 heartbeater.cc:511] Master 127.0.202.190:37521 requested a full tablet report, sending...
I20260927 21:40:11.560675 9143 ts_manager.cc:194] Registered new tserver with Master: 1668b497740a4b88a9db3efe74a890bb (127.0.202.129:41457)
I20260927 21:40:11.562105 9143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:55209
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:40:11.595661 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:37521
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:11.927841 9338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:11.928216 9338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:11.928529 9338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:11.936549 9338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:11.936699 9338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:11.936870 9338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:40:11.946815 9338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:11.948135 9338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:11.949510 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:11.963068 9344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:11.963020 9346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:11.963030 9343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:11.963994 9338 server_base.cc:1034] running on GCE node
I20260927 21:40:11.964768 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:11.965857 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:11.967118 9338 hybrid_clock.cc:648] HybridClock initialized: now 1790545211967084 us; error 46 us; skew 500 ppm
I20260927 21:40:11.967495 9338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:11 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:11.970800 9338 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:40:11.972383 9338 webserver.cc:460] Webserver started at http://127.0.202.130:46767/ using document root <none> and password file <none>
I20260927 21:40:11.973133 9338 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:11.973290 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:11.973613 9338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:11.976673 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "2ffa121e91504cf89d4f8d9725795a47"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "eca1335d383e332be40271ba15dbb8f0"
server_key_iv: "c41544f2f7a7ec5012e5f79df9564ae2"
server_key_version: "encryptionkey@0"
I20260927 21:40:11.977473 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "2ffa121e91504cf89d4f8d9725795a47"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "eca1335d383e332be40271ba15dbb8f0"
server_key_iv: "c41544f2f7a7ec5012e5f79df9564ae2"
server_key_version: "encryptionkey@0"
I20260927 21:40:11.982784 9338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:40:11.986539 9353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:11.988166 9338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 21:40:11.988488 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "2ffa121e91504cf89d4f8d9725795a47"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "eca1335d383e332be40271ba15dbb8f0"
server_key_iv: "c41544f2f7a7ec5012e5f79df9564ae2"
server_key_version: "encryptionkey@0"
I20260927 21:40:11.988796 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:12.008873 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:12.013545 9338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:12.013875 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:12.014932 9338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:12.016378 9338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:12.016484 9338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:12.016647 9338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:12.016722 9338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:12.051563 9338 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:33265
I20260927 21:40:12.051803 9465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:33265 every 8 connection(s)
I20260927 21:40:12.053010 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:40:12.058379 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9338
I20260927 21:40:12.058575 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:40:12.059154 810 external_mini_cluster.cc:1428] Setting key c68b197712141901ce285b903ff192da
Sep 27 21:40:12 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:12.079734 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:12.055910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:32807 (local address 127.0.202.190:37521)
0927 21:40:12.056396 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:12.056403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:12.057425 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 21:40:12.060110 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:12.060115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:12.060184 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:12.060335 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:12.062418 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.063088 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.065839 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.066324 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.066433 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:12.071352 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:12.071362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:12.071371 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:12.071424 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:12.075699 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.076670 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.076680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.076687 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.076858 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.077365 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.077373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.077380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.077868 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:12.078031 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:12.078871 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 21:40:12.079174 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":202,"threads_started":1}
I20260927 21:40:12.080623 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:12.056521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:32807)
0927 21:40:12.057236 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:12.057291 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:12.059807 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:12.060581 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:12.060590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:12.061328 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:12.062194 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:12.062222 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.063254 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.063263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:12.065643 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:12.065661 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.066572 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.066582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:12.066945 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:12.067729 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:12.067781 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:12.071040 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:12.075900 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:12.075910 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:12.075925 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:12.076517 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:12.077019 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:12.077023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:12.077025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:12.077223 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:12.078012 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:12.078019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:12.078442 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 21:40:12.078711 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 21:40:12.079252 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":273,"threads_started":1}
I20260927 21:40:12.082230 9466 heartbeater.cc:347] Connected to a master server at 127.0.202.190:37521
I20260927 21:40:12.082628 9466 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:12.083693 9466 heartbeater.cc:511] Master 127.0.202.190:37521 requested a full tablet report, sending...
I20260927 21:40:12.085475 9143 ts_manager.cc:194] Registered new tserver with Master: 2ffa121e91504cf89d4f8d9725795a47 (127.0.202.130:33265)
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:40:12.086387 9143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:32807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:40:12.116611 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:37521
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:12.460142 9472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:12.460497 9472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:12.460597 9472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:12.467839 9472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:12.467978 9472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:12.468096 9472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:40:12.478091 9472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:12.479509 9472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:12.480865 9472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:12.494741 9477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:12.494706 9480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:12.494706 9478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:12.495536 9472 server_base.cc:1034] running on GCE node
I20260927 21:40:12.496177 9472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:12.497263 9472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:12.498499 9472 hybrid_clock.cc:648] HybridClock initialized: now 1790545212498436 us; error 54 us; skew 500 ppm
I20260927 21:40:12.498847 9472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:12 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:12.502456 9472 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:40:12.504150 9472 webserver.cc:460] Webserver started at http://127.0.202.131:34219/ using document root <none> and password file <none>
I20260927 21:40:12.504978 9472 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:12.505127 9472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:12.505438 9472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:12.509250 9472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "28ad0301184e4224a3793216fc50bb7f"
format_stamp: "Formatted at 2026-09-27 21:40:12 on dist-test-slave-25dq"
server_key: "75773a8e2c1004b315ce6da508d66ed5"
server_key_iv: "ee3774836fb35e69560bfcad01cf4176"
server_key_version: "encryptionkey@0"
I20260927 21:40:12.510272 9472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "28ad0301184e4224a3793216fc50bb7f"
format_stamp: "Formatted at 2026-09-27 21:40:12 on dist-test-slave-25dq"
server_key: "75773a8e2c1004b315ce6da508d66ed5"
server_key_iv: "ee3774836fb35e69560bfcad01cf4176"
server_key_version: "encryptionkey@0"
I20260927 21:40:12.517105 9472 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 21:40:12.521121 9487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:12.522737 9472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260927 21:40:12.522921 9472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "28ad0301184e4224a3793216fc50bb7f"
format_stamp: "Formatted at 2026-09-27 21:40:12 on dist-test-slave-25dq"
server_key: "75773a8e2c1004b315ce6da508d66ed5"
server_key_iv: "ee3774836fb35e69560bfcad01cf4176"
server_key_version: "encryptionkey@0"
I20260927 21:40:12.523110 9472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:12.538970 9472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:12.543776 9472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:12.544145 9472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:12.545236 9472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:12.546777 9472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:12.546854 9472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:12.547008 9472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:12.547087 9472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:12.565845 9332 heartbeater.cc:503] Master 127.0.202.190:37521 was elected leader, sending a full tablet report...
I20260927 21:40:12.581817 9472 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37905
I20260927 21:40:12.581864 9599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37905 every 8 connection(s)
I20260927 21:40:12.583145 9472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:40:12.590098 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9472
I20260927 21:40:12.590276 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:40:12.590737 810 external_mini_cluster.cc:1428] Setting key 5f5d10a4063a2e993fe4478f22fc44ff
Sep 27 21:40:12 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:12.606747 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:12.586014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:35603 (local address 127.0.202.190:37521)
0927 21:40:12.586485 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:12.586490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:12.587533 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 21:40:12.589559 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:12.589563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:12.589632 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:12.589785 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:12.591708 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.593001 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.595046 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.595712 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.595853 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:12.599984 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:12.599990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:12.599995 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:12.600037 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:12.602897 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.603711 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.603718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.603725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.603869 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.604372 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.604379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.604383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.604771 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:12.605010 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:12.606037 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0927 21:40:12.606288 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":225,"threads_started":1}
I20260927 21:40:12.607854 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:12.586619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:35603)
0927 21:40:12.587360 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:12.587403 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:12.589290 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:12.590052 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:12.590061 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:12.590731 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:12.591517 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:12.591539 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.593197 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.593202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:12.594897 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:12.594909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.596024 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.596035 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:12.596416 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:12.597445 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:12.597467 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:12.599741 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:12.603081 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:12.603092 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:12.603112 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:12.603571 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:12.603990 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:12.603996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:12.604000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:12.604204 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:12.604993 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:12.605002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:12.605494 (+ 492us) client_negotiation.cc:769] Sending connection context
0927 21:40:12.605885 (+ 391us) client_negotiation.cc:240] Negotiation successful
0927 21:40:12.606418 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":303,"threads_started":1}
I20260927 21:40:12.609521 9600 heartbeater.cc:347] Connected to a master server at 127.0.202.190:37521
I20260927 21:40:12.609900 9600 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:12.610683 9600 heartbeater.cc:511] Master 127.0.202.190:37521 requested a full tablet report, sending...
I20260927 21:40:12.612229 9143 ts_manager.cc:194] Registered new tserver with Master: 28ad0301184e4224a3793216fc50bb7f (127.0.202.131:37905)
I20260927 21:40:12.613134 9143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:35603
I20260927 21:40:12.620026 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab.
Entry for principal oryx/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab.
WARNING: no policy specified for oryx/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab.
Entry for principal oryx/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab.
WARNING: no policy specified for oryx/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab.
Entry for principal oryx/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab.
I20260927 21:40:12.780406 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9204
Sep 27 21:40:12 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:40:12.799265 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:12.781650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.202.129:41457)
0927 21:40:12.782272 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:12.782280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:12.782311 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:40:12.782424 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:12.782431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:12.782651 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:12.782788 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:12.784389 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.785547 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.787248 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.787872 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.787999 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:12.792764 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:12.792791 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:12.792808 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:12.792854 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:12.796139 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.796987 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.796996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.796998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.797144 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.797622 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.797625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.797627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.798008 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:12.798247 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:12.798594 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 21:40:12.798883 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":346,"threads_started":1}
I20260927 21:40:12.800762 9327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:12.885000 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9204
I20260927 21:40:12.902601 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9338
Sep 27 21:40:12 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:40:12.920243 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:12.903504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.0.202.130:33265)
0927 21:40:12.904093 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:12.904100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:12.904127 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:40:12.904198 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:12.904205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:12.904520 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:12.904713 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:12.906054 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.907150 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.908634 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:12.909229 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:12.909393 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:12.913942 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:12.913954 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:12.913969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:12.914005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:12.917168 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.917939 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.917948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.917951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.918086 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:12.918491 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:12.918499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:12.918502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:12.918865 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:12.919202 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:12.919574 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 21:40:12.919860 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":305,"threads_started":1}
I20260927 21:40:12.922011 9461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:13.010130 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9338
I20260927 21:40:13.027309 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9472
Sep 27 21:40:13 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:40:13.046057 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:13.028562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.0.202.131:37905)
0927 21:40:13.029125 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:13.029133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:13.029170 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:40:13.029241 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:13.029248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:13.029479 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:13.029678 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:13.031309 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:13.032530 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:13.034027 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:13.034707 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:13.034894 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:13.039851 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:13.039863 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:13.039878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:13.039915 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:13.043197 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:13.043986 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:13.043993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:13.043995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:13.044127 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:13.044616 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:13.044620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:13.044622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:13.045015 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:13.045248 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:13.045549 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:40:13.045745 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":273,"threads_started":1}
I20260927 21:40:13.047380 9595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:13.132486 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9472
I20260927 21:40:13.149154 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9111
I20260927 21:40:13.150484 9173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:13.254168 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9111
I20260927 21:40:13.275249 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:37521
--webserver_interface=127.0.202.190
--webserver_port=43031
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:37521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:13.631182 9618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:13.631512 9618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:13.631678 9618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:13.639106 9618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:40:13.639238 9618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:13.639333 9618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:13.639422 9618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:40:13.639513 9618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:40:13.649886 9618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:37521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:37521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=43031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:13.651283 9618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:13.652628 9618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:13.664842 9623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:13.664904 9624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:13.664943 9626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:13.665731 9618 server_base.cc:1034] running on GCE node
I20260927 21:40:13.666564 9618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:13.668140 9618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:13.669379 9618 hybrid_clock.cc:648] HybridClock initialized: now 1790545213669318 us; error 82 us; skew 500 ppm
I20260927 21:40:13.669821 9618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:13 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:13.672993 9618 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:40:13.674438 9618 webserver.cc:460] Webserver started at http://127.0.202.190:43031/ using document root <none> and password file <none>
I20260927 21:40:13.675092 9618 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:13.675201 9618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:13.681006 9618 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:40:13.684024 9633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:13.685482 9618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:13.685631 9618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "c4711d71f16c4891a6e6b814249fba15"
format_stamp: "Formatted at 2026-09-27 21:40:10 on dist-test-slave-25dq"
server_key: "6ae9ed4687a2e8f0a1282c2eda8b6270"
server_key_iv: "d12570caa674784904db38902c9d281e"
server_key_version: "encryptionkey@0"
I20260927 21:40:13.686594 9618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:13.711009 9618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:13.715509 9618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:13.715736 9618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:13.734368 9618 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:37521
I20260927 21:40:13.734387 9684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:37521 every 8 connection(s)
I20260927 21:40:13.735877 9618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:40:13.739084 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9618
I20260927 21:40:13.740226 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:41457
--local_ip_for_outbound_sockets=127.0.202.129
--tserver_master_addrs=127.0.202.190:37521
--webserver_port=41423
--webserver_interface=127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:40:13.746922 9685 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:40:13.751405 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Bootstrap starting.
I20260927 21:40:13.773167 9685 log.cc:824] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:13.783088 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:40:13.783725 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Bootstrap complete.
I20260927 21:40:13.795203 9685 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:13.796089 9685 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4711d71f16c4891a6e6b814249fba15, State: Initialized, Role: FOLLOWER
I20260927 21:40:13.796896 9685 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:13.797142 9685 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:40:13.797260 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:40:13.797451 9685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:40:13.801415 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:13.801959 9685 leader_election.cc:302] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4711d71f16c4891a6e6b814249fba15; no voters:
I20260927 21:40:13.802665 9685 leader_election.cc:288] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:40:13.802814 9689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:40:13.803917 9689 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [term 2 LEADER]: Becoming Leader. State: Replica: c4711d71f16c4891a6e6b814249fba15, State: Running, Role: LEADER
I20260927 21:40:13.804569 9689 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } }
I20260927 21:40:13.804999 9685 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:40:13.807053 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4711d71f16c4891a6e6b814249fba15. Latest consensus state: current_term: 2 leader_uuid: "c4711d71f16c4891a6e6b814249fba15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } } }
I20260927 21:40:13.807438 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:13.807966 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c4711d71f16c4891a6e6b814249fba15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4711d71f16c4891a6e6b814249fba15" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37521 } } }
I20260927 21:40:13.808391 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:13.812688 9695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:40:13.820704 9695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:40:13.822281 9695 catalog_manager.cc:1259] Loaded cluster ID: 4b27689abd9940b98d1fb63b2aac3adf
I20260927 21:40:13.822465 9695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:40:13.826709 9695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:40:13.828431 9695 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4711d71f16c4891a6e6b814249fba15: Loaded TSK: 0
I20260927 21:40:13.830118 9695 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:40:14.091988 9687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:14.092276 9687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:14.092474 9687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:14.099879 9687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:14.099996 9687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:14.100171 9687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:40:14.110515 9687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.129.keytab
--principal=oryx/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:41457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=41423
--tserver_master_addrs=127.0.202.190:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:14.111922 9687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:14.113427 9687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:14.127326 9711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:14.127389 9712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:14.128026 9714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:14.128320 9687 server_base.cc:1034] running on GCE node
I20260927 21:40:14.128947 9687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:14.130003 9687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:14.131237 9687 hybrid_clock.cc:648] HybridClock initialized: now 1790545214131175 us; error 58 us; skew 500 ppm
I20260927 21:40:14.131572 9687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:14 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:14.134877 9687 init.cc:377] Logged in from keytab as oryx/127.0.202.129@KRBTEST.COM (short username oryx)
I20260927 21:40:14.136844 9687 webserver.cc:460] Webserver started at http://127.0.202.129:41423/ using document root <none> and password file <none>
I20260927 21:40:14.137643 9687 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:14.137795 9687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:14.144193 9687 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:40:14.147464 9721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:14.149209 9687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:40:14.149432 9687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "1668b497740a4b88a9db3efe74a890bb"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "007c6a478c332a7f1e064a38fb91ae6e"
server_key_iv: "1f25b7ce425ce98047db2039fa4bc341"
server_key_version: "encryptionkey@0"
I20260927 21:40:14.150384 9687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:14.166990 9687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:14.171656 9687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:14.171953 9687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:14.173101 9687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:14.174487 9687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:14.174615 9687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:14.174736 9687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:14.174822 9687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:14.215533 9687 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41457
I20260927 21:40:14.215705 9833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41457 every 8 connection(s)
I20260927 21:40:14.217104 9687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:40:14.226603 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9687
I20260927 21:40:14.227262 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:33265
--local_ip_for_outbound_sockets=127.0.202.130
--tserver_master_addrs=127.0.202.190:37521
--webserver_port=46767
--webserver_interface=127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:40:14 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:14 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:40:14.571682 9838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:14.571960 9838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:14.572098 9838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:14.579763 9838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:14.579881 9838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:14.580054 9838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:40:14.590353 9838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.130.keytab
--principal=oryx/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:33265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=46767
--tserver_master_addrs=127.0.202.190:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:14.591805 9838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:14.593207 9838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:14.612035 9847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:14.612226 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:14.613618 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:14.614099 9838 server_base.cc:1034] running on GCE node
I20260927 21:40:14.614845 9838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:14.616020 9838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:14.617260 9838 hybrid_clock.cc:648] HybridClock initialized: now 1790545214617184 us; error 70 us; skew 500 ppm
I20260927 21:40:14.617682 9838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:14 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:14.621623 9838 init.cc:377] Logged in from keytab as oryx/127.0.202.130@KRBTEST.COM (short username oryx)
I20260927 21:40:14.623452 9838 webserver.cc:460] Webserver started at http://127.0.202.130:46767/ using document root <none> and password file <none>
I20260927 21:40:14.624270 9838 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:14.624517 9838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:14.631791 9838 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 21:40:14.637224 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:14.639379 9838 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:40:14.639632 9838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "2ffa121e91504cf89d4f8d9725795a47"
format_stamp: "Formatted at 2026-09-27 21:40:11 on dist-test-slave-25dq"
server_key: "eca1335d383e332be40271ba15dbb8f0"
server_key_iv: "c41544f2f7a7ec5012e5f79df9564ae2"
server_key_version: "encryptionkey@0"
I20260927 21:40:14.640725 9838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:14.679415 9838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:14.683892 9838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:14.684178 9838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:14.685289 9838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:14.686685 9838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:14.686775 9838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:14.686930 9838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:14.687002 9838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:14.729429 9838 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:33265
I20260927 21:40:14.729483 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:33265 every 8 connection(s)
I20260927 21:40:14.730893 9838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:40:14.731799 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9838
I20260927 21:40:14.732712 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:37905
--local_ip_for_outbound_sockets=127.0.202.131
--tserver_master_addrs=127.0.202.190:37521
--webserver_port=34219
--webserver_interface=127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:40:14 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:14 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:40:15.461959 9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:15.462587 9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:15.462826 9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:15.478538 9837 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.247s user 0.005s sys 0.008s
W20260927 21:40:15.480575 9836 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:14.220386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:49671 (local address 127.0.202.190:37521)
0927 21:40:14.221317 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:14.221336 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:14.221625 (+ 289us) server_negotiation.cc:407] Connection header received
0927 21:40:14.223998 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:14.224012 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:14.224448 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:14.224839 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:14.226492 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.228086 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.229825 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.230601 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.230826 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.479713 (+1248887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:49671: BlockingRecv error: recv got EOF from 127.0.202.129:49671 (error 108)
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":367,"threads_started":1}
I20260927 21:40:15.480571 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:14.220471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:49671)
0927 21:40:14.221458 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:14.221498 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:14.223350 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:14.224987 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:14.225003 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:14.225660 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:14.226277 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:14.226294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.228256 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.228262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:14.229668 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:14.229680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.230855 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.230864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:14.231161 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:14.231918 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:14.231942 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:15.479344 (+1247402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":380,"threads_started":1}
W20260927 21:40:15.481885 9971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:15.481912 9834 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:37521 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.202.190:37521: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
W20260927 21:40:15.482033 9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:15.482225 9971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:15.496701 9971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/krb5kdc/oryx_127.0.202.131.keytab
--principal=oryx/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:37905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=34219
--tserver_master_addrs=127.0.202.190:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:15.498162 9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:15.498193 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:15.482623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:36805)
0927 21:40:15.483052 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:15.483076 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:15.483206 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:15.483718 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:15.483723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:15.484358 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:15.484761 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.484775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.487545 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.487553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.489175 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.489193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.491001 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.491008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.491187 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.492095 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:15.492118 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:15.497675 (+ 5557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144}
W20260927 21:40:15.499528 9836 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:15.483003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:36805 (local address 127.0.202.190:37521)
0927 21:40:15.483246 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:15.483255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:15.483286 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:40:15.483380 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:15.483389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:15.483500 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:15.483685 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:15.485012 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.487250 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.489415 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.490751 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.490975 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.499099 (+ 8124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:36805: BlockingRecv error: recv got EOF from 127.0.202.129:36805 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:40:15.500007 9971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:40:15.521526 9836 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:15.499428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:49479 (local address 127.0.202.190:37521)
0927 21:40:15.499883 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:15.499894 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:15.500989 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 21:40:15.501093 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:15.501102 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:15.501259 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:15.501395 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:15.503086 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.505096 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.508812 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.510099 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.510273 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.521033 (+ 10760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:49479: BlockingRecv error: recv got EOF from 127.0.202.129:49479 (error 108)
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":11271,"threads_started":1}
I20260927 21:40:15.522670 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:15.499094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:49479)
0927 21:40:15.500586 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:15.500619 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:15.500818 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:15.501696 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:15.501706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:15.502416 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:15.502781 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.502802 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.505577 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.505588 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.508513 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.508543 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.511180 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.511192 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.511592 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.512858 (+ 1266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:15.512953 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:15.522183 (+ 9230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1294}
W20260927 21:40:15.523480 9834 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:40:15.523010 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:15.525167 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:15.530150 9982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:15.530592 9971 server_base.cc:1034] running on GCE node
I20260927 21:40:15.531905 9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:15.533880 9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:15.535229 9971 hybrid_clock.cc:648] HybridClock initialized: now 1790545215535147 us; error 50 us; skew 500 ppm
I20260927 21:40:15.535866 9971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545215, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:15.542059 9971 init.cc:377] Logged in from keytab as oryx/127.0.202.131@KRBTEST.COM (short username oryx)
I20260927 21:40:15.544421 9971 webserver.cc:460] Webserver started at http://127.0.202.131:34219/ using document root <none> and password file <none>
I20260927 21:40:15.545234 9971 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:15.545418 9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:15.551617 9971 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 21:40:15.555141 9988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:15.557128 9971 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 21:40:15.557322 9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "28ad0301184e4224a3793216fc50bb7f"
format_stamp: "Formatted at 2026-09-27 21:40:12 on dist-test-slave-25dq"
server_key: "75773a8e2c1004b315ce6da508d66ed5"
server_key_iv: "ee3774836fb35e69560bfcad01cf4176"
server_key_version: "encryptionkey@0"
I20260927 21:40:15.558283 9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:15.578094 9971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:15.583467 9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:15.583791 9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:15.585057 9971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:15.586642 9971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:15.586777 9971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:15.586927 9971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:15.587028 9971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:15.629952 9971 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37905
I20260927 21:40:15.629997 10100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37905 every 8 connection(s)
I20260927 21:40:15.631525 9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:40:15.632211 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 9971
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:15.656646 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:15.633912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.0.202.190:37521)
0927 21:40:15.634178 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:15.634188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:15.636925 (+ 2737us) server_negotiation.cc:407] Connection header received
0927 21:40:15.637063 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:15.637073 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:15.637212 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:15.637388 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:15.640956 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.642073 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.645209 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.645775 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.645924 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.650173 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:15.650181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:15.650199 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:15.650236 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:15.653155 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:15.654042 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:15.654049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:15.654057 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:15.654244 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:15.654663 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:15.654674 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:15.654677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:15.655188 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:15.655509 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:15.655948 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 21:40:15.656389 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 21:40:15.961479 9970 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.203s user 0.005s sys 0.008s
I20260927 21:40:15.963284 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:14.735742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:46561)
0927 21:40:14.745920 (+ 10178us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:14.745979 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:14.748597 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:14.752241 (+ 3644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:14.752250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:14.752846 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:14.753539 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:14.753574 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.754795 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.754802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:14.756061 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:14.756072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.756980 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.756988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:14.757229 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:14.757954 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:14.757977 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:15.962305 (+1204328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9839,"thread_start_us":9579,"threads_started":1}
W20260927 21:40:15.963433 9969 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:14.734841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:46561 (local address 127.0.202.190:37521)
0927 21:40:14.751662 (+ 16821us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:14.751667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:14.751684 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:40:14.751733 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:14.751736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:14.751807 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:14.751953 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:14.753739 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.754510 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.756217 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:14.756705 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:14.756845 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.962766 (+1205921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:46561: BlockingRecv error: recv got EOF from 127.0.202.130:46561 (error 108)
Metrics: {"server-negotiator.queue_time_us":16709,"thread_start_us":17233,"threads_started":1}
W20260927 21:40:15.964679 9967 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:37521 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.202.190:37521: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:15.983290 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:15.965642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:51941)
0927 21:40:15.965827 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:15.965853 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:15.965984 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:15.966500 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:15.966504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:15.967031 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:15.967263 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.967277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.969676 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.969680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.970804 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.970816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.972323 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.972334 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.972487 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.978299 (+ 5812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:15.978327 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:15.983054 (+ 4727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:40:15.983901 9969 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:15.965799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:51941 (local address 127.0.202.190:37521)
0927 21:40:15.966016 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:15.966025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:15.966048 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:40:15.966115 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:15.966120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:15.966201 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:15.966311 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:15.967450 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.969438 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.970974 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.972028 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.972217 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.983490 (+ 11273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:51941: BlockingRecv error: recv got EOF from 127.0.202.130:51941 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:15 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:15.994867 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:15.984080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:38585)
0927 21:40:15.984424 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:15.984445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:15.984573 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:15.984935 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:15.984941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:15.985286 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:15.985442 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.985451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.986596 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.986602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.987672 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.987684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.988530 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.988536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.988739 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.989450 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:15.989486 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:15.994652 (+ 5166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236}
W20260927 21:40:15.995240 9967 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:40:15.995541 9836 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:15.984232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:38585 (local address 127.0.202.190:37521)
0927 21:40:15.984426 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:15.984434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:15.984510 (+ 76us) server_negotiation.cc:407] Connection header received
0927 21:40:15.984708 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:15.984714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:15.984793 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:15.984901 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:15.985597 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.986447 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.987827 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.988346 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.988526 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.995099 (+ 6573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:38585: BlockingRecv error: recv got EOF from 127.0.202.130:38585 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:16.538772 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:16.524268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:53991)
0927 21:40:16.525104 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:16.525133 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:16.525317 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:16.526308 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:16.526312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:16.526760 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:16.526912 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.526920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.529016 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.529021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:16.530811 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.530838 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.532148 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.532153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:16.532285 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:16.533549 (+ 1264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:16.533575 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:16.538249 (+ 4674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":446,"threads_started":1}
W20260927 21:40:16.538839 10107 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:16.524329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:53991 (local address 127.0.202.190:37521)
0927 21:40:16.525897 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:16.525905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:16.525928 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:40:16.526000 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:16.526009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:16.526115 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:16.526263 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:16.527073 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.528829 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.531025 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.531889 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.532694 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:16.538627 (+ 5933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:53991: BlockingRecv error: recv got EOF from 127.0.202.129:53991 (error 108)
Metrics: {"server-negotiator.queue_time_us":1413,"thread_start_us":1282,"threads_started":1}
W20260927 21:40:16.815310 10104 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.169s user 0.000s sys 0.026s
W20260927 21:40:16.816918 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:15.637001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:47801 (local address 127.0.202.190:37521)
0927 21:40:15.637192 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:15.637197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:15.637500 (+ 303us) server_negotiation.cc:407] Connection header received
0927 21:40:15.639494 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:15.639498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:15.639575 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:15.639671 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:15.641436 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.642454 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.643998 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.644534 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.644669 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:16.816647 (+1171978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:47801: BlockingRecv error: recv got EOF from 127.0.202.131:47801 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:40:16.817559 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:15.636381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:47801)
0927 21:40:15.637334 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:15.637396 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:15.639208 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:15.639939 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:15.639948 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:15.640525 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:15.641215 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.641237 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.642600 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.642603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.643853 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:15.643864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:15.644760 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:15.644770 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:15.645019 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:15.645965 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:15.646004 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:16.816370 (+1170366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":690,"threads_started":1}
W20260927 21:40:16.818521 10101 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:37521 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.202.190:37521: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:16.829051 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:16.819170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:56305)
0927 21:40:16.819393 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:16.819417 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:16.819521 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:16.819884 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:16.819887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:16.820232 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:16.820519 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.820529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.821610 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.821614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:16.822617 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.822629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.823494 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.823500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:16.823623 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:16.824258 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:16.824278 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:16.828739 (+ 4461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 21:40:16.829154 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:16.819306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:56305 (local address 127.0.202.190:37521)
0927 21:40:16.819610 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:16.819615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:16.819632 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:40:16.819678 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:16.819682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:16.819755 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:16.819846 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:16.820655 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.821447 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.822757 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.823254 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.823363 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:16.828936 (+ 5573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:56305: BlockingRecv error: recv got EOF from 127.0.202.131:56305 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:16.838790 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:16.829860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:43497)
0927 21:40:16.830043 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:16.830064 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:16.830161 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:16.830578 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:16.830582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:16.830862 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:16.831019 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.831026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.832123 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.832128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:16.833252 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.833265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.833997 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.834001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:16.834124 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:16.834765 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:16.834783 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:16.838560 (+ 3777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 21:40:16.839059 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:16.830106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:43497 (local address 127.0.202.190:37521)
0927 21:40:16.830301 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:16.830306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:16.830321 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:40:16.830363 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:16.830367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:16.830425 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:16.830510 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:16.831147 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.831971 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.833403 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.833838 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.833960 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:16.838789 (+ 4829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:43497: BlockingRecv error: recv got EOF from 127.0.202.131:43497 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 21:40:16.839223 10101 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:16 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:17.006093 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:16.996025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:41713)
0927 21:40:16.996783 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:16.996803 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:16.997002 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:16.997364 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:16.997369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:16.997810 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:16.997949 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.997957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.998940 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:16.998947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:16.999929 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:16.999943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.000745 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.000751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:17.000921 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:17.001570 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:17.001587 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:17.005663 (+ 4076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":487,"threads_started":1}
W20260927 21:40:17.006130 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:16.996500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:41713 (local address 127.0.202.190:37521)
0927 21:40:16.996752 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:16.996758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:16.996894 (+ 136us) server_negotiation.cc:407] Connection header received
0927 21:40:16.997138 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:16.997143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:16.997228 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:16.997334 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:16.998088 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:16.998797 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.000071 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.000561 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.000689 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:17.005856 (+ 5167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:41713: BlockingRecv error: recv got EOF from 127.0.202.130:41713 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 21:40:17 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:17 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:17.549966 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:17.539991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:52373)
0927 21:40:17.540572 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:17.540597 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:17.540746 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:17.541165 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:17.541170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:17.541779 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:17.541966 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:17.541976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.543000 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.543006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:17.544047 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:17.544060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.544942 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.544947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:17.545097 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:17.545847 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:17.545866 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:17.549708 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":310,"threads_started":1}
W20260927 21:40:17.550140 10110 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:17.540002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:52373 (local address 127.0.202.190:37521)
0927 21:40:17.540610 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:17.540617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:17.540661 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:40:17.540887 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:17.540892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:17.540972 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:17.541115 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:17.542104 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.542851 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.544191 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.544805 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.544898 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:17.549913 (+ 5015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:52373: BlockingRecv error: recv got EOF from 127.0.202.129:52373 (error 108)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":299,"threads_started":1}
Sep 27 21:40:17 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:17 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:17.850919 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:17.839881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:44195)
0927 21:40:17.841229 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:17.841254 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:17.841395 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:17.841800 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:17.841805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:17.842424 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:17.842628 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:17.842640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.843730 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.843735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:17.844832 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:17.844843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.845796 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.845801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:17.845986 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:17.846785 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:17.846805 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:17.850682 (+ 3877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":962,"threads_started":1}
W20260927 21:40:17.851248 10110 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:17.840083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:44195 (local address 127.0.202.190:37521)
0927 21:40:17.840550 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:17.840558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:17.841337 (+ 779us) server_negotiation.cc:407] Connection header received
0927 21:40:17.841549 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:17.841555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:17.841632 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:17.841729 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:17.842794 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.843579 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.844996 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:17.845600 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:17.845706 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:17.850922 (+ 5216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:44195: BlockingRecv error: recv got EOF from 127.0.202.131:44195 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Sep 27 21:40:18 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:18 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:18.017130 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:18.007296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:40809)
0927 21:40:18.007877 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:18.007898 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:18.008032 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:18.008518 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:18.008525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:18.008972 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:18.009150 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:18.009158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.010216 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.010225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:18.011214 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:18.011227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.011957 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.011963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:18.012146 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:18.012872 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:18.012891 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:18.016744 (+ 3853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":336,"threads_started":1}
W20260927 21:40:18.017244 10110 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:18.007390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:40809 (local address 127.0.202.190:37521)
0927 21:40:18.007703 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:18.007711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:18.008022 (+ 311us) server_negotiation.cc:407] Connection header received
0927 21:40:18.008128 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:18.008133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:18.008215 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:18.008410 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:18.009299 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.010063 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.011365 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.011794 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.011941 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:18.016971 (+ 5030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:40809: BlockingRecv error: recv got EOF from 127.0.202.130:40809 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 27 21:40:18 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:18 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:18.561178 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:18.550986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:59391)
0927 21:40:18.551622 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:18.551647 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:18.551765 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:18.552147 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:18.552152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:18.552671 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:18.552857 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:18.552866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.554042 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.554047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:18.555475 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:18.555491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.556285 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.556339 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:18.556576 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:18.557214 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:18.557231 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:18.560830 (+ 3599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":320,"threads_started":1}
W20260927 21:40:18.561328 10114 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:18.551173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:59391 (local address 127.0.202.190:37521)
0927 21:40:18.551735 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:18.551740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:18.551760 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:40:18.551886 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:18.551891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:18.551974 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:18.552083 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:18.553010 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.553900 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.555634 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.556133 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.556241 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:18.561055 (+ 4814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:59391: BlockingRecv error: recv got EOF from 127.0.202.129:59391 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":334,"threads_started":1}
Sep 27 21:40:18 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:18 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:18.862739 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:18.852051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:38573)
0927 21:40:18.852755 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:18.852773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:18.852891 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:18.853226 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:18.853230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:18.853688 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:18.853880 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:18.853890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.855180 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.855185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:18.856627 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:18.856641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.857350 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.857356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:18.857574 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:18.858216 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:18.858235 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:18.862307 (+ 4072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":467,"threads_started":1}
W20260927 21:40:18.862766 10114 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:18.852105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:38573 (local address 127.0.202.190:37521)
0927 21:40:18.852317 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:18.852325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:18.852856 (+ 531us) server_negotiation.cc:407] Connection header received
0927 21:40:18.852991 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:18.852994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:18.853061 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:18.853150 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:18.854071 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.854875 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.856776 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:18.857175 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:18.857282 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:18.862550 (+ 5268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:38573: BlockingRecv error: recv got EOF from 127.0.202.131:38573 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:40:19 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:19 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:19.027956 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:19.018462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:58237)
0927 21:40:19.019012 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:19.019030 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:19.019172 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:19.019563 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:19.019567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:19.019946 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:19.020117 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:19.020126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.021184 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.021188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:19.022108 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:19.022118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.022855 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.022859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:19.022979 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:19.023565 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:19.023590 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:19.027525 (+ 3935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
W20260927 21:40:19.027962 10114 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:19.018556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:58237 (local address 127.0.202.190:37521)
0927 21:40:19.018848 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:19.018856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:19.019142 (+ 286us) server_negotiation.cc:407] Connection header received
0927 21:40:19.019327 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:19.019332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:19.019405 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:19.019505 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:19.020274 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.021066 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.022246 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.022725 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.022826 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:19.027721 (+ 4895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:58237: BlockingRecv error: recv got EOF from 127.0.202.130:58237 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 21:40:19 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:19 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:19.572784 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:19.562336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:50279)
0927 21:40:19.562907 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:19.562955 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:19.563078 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:19.563627 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:19.563631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:19.563981 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:19.564151 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:19.564159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.565502 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.565509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:19.566853 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:19.566865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.567701 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.567709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:19.567855 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:19.568691 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:19.568711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:19.572434 (+ 3723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":295,"threads_started":1}
W20260927 21:40:19.572918 10118 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:19.562629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:50279 (local address 127.0.202.190:37521)
0927 21:40:19.563266 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:19.563271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:19.563288 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:40:19.563337 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:19.563342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:19.563400 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:19.563496 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:19.564344 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.565360 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.566992 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.567441 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.567621 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:19.572673 (+ 5052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:50279: BlockingRecv error: recv got EOF from 127.0.202.129:50279 (error 108)
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":405,"threads_started":1}
Sep 27 21:40:19 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:19 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:19.873674 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:19.863766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:58929)
0927 21:40:19.864413 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:19.864428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:19.864554 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:19.864920 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:19.864923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:19.865281 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:19.865442 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:19.865450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.866583 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.866587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:19.867525 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:19.867536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.868326 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.868331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:19.868445 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:19.869157 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:19.869184 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:19.873334 (+ 4150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":378,"threads_started":1}
W20260927 21:40:19.873750 10118 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:19.863865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:58929 (local address 127.0.202.190:37521)
0927 21:40:19.864058 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:19.864063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:19.864532 (+ 469us) server_negotiation.cc:407] Connection header received
0927 21:40:19.864706 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:19.864710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:19.864781 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:19.864877 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:19.865647 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.866441 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.867712 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:19.868153 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:19.868277 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:19.873528 (+ 5251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:58929: BlockingRecv error: recv got EOF from 127.0.202.131:58929 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 21:40:20 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:20 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:20.039717 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:20.029337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:56139)
0927 21:40:20.029908 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:20.029926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:20.030052 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:20.030605 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:20.030610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:20.031068 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:20.031241 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:20.031251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.032426 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.032431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:20.033447 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:20.033459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.034166 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.034170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:20.034298 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:20.035086 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:20.035104 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:20.039347 (+ 4243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":327,"threads_started":1}
W20260927 21:40:20.039829 10118 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:20.029399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:56139 (local address 127.0.202.190:37521)
0927 21:40:20.029642 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:20.029647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:20.030040 (+ 393us) server_negotiation.cc:407] Connection header received
0927 21:40:20.030238 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:20.030243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:20.030341 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:20.030497 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:20.031410 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.032192 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.033592 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.034018 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.034119 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:20.039575 (+ 5456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:56139: BlockingRecv error: recv got EOF from 127.0.202.130:56139 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 21:40:20 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:20 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:20.583869 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:20.573880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:42637)
0927 21:40:20.574471 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:20.574489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:20.574613 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:20.574957 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:20.574960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:20.575424 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:20.575581 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:20.575591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.576736 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.576741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:20.577819 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:20.577833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.578595 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.578600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:20.578733 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:20.579568 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:20.579586 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:20.583482 (+ 3896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":349,"threads_started":1}
W20260927 21:40:20.583953 10122 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:20.573968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:42637 (local address 127.0.202.190:37521)
0927 21:40:20.574434 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:20.574439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:20.574600 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:40:20.574731 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:20.574735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:20.574817 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:20.574909 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:20.575727 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.576587 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.577974 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.578424 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.578529 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:20.583694 (+ 5165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:42637: BlockingRecv error: recv got EOF from 127.0.202.129:42637 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":252,"threads_started":1}
Sep 27 21:40:20 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:20 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:20.884991 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:20.874843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:59963)
0927 21:40:20.875483 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:20.875500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:20.875616 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:20.875949 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:20.875952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:20.876428 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:20.876611 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:20.876620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.877708 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.877713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:20.878905 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:20.878922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.879606 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.879611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:20.879752 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:20.880536 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:20.880556 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:20.884747 (+ 4191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":392,"threads_started":1}
W20260927 21:40:20.885251 10122 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:20.874934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:59963 (local address 127.0.202.190:37521)
0927 21:40:20.875259 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:20.875267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:20.875610 (+ 343us) server_negotiation.cc:407] Connection header received
0927 21:40:20.875716 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:20.875720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:20.875793 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:20.875880 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:20.876757 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.877505 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.879066 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:20.879445 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:20.879541 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:20.884957 (+ 5416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:59963: BlockingRecv error: recv got EOF from 127.0.202.131:59963 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 27 21:40:21 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:21 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:21.057824 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:21.040827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:54511)
0927 21:40:21.041425 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:21.041443 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:21.041580 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:21.042199 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:21.042203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:21.042642 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:21.042805 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:21.042813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.044256 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.044262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:21.045874 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:21.045888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.046706 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.046711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:21.046837 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:21.053451 (+ 6614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:21.053474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:21.057490 (+ 4016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":333,"threads_started":1}
W20260927 21:40:21.058017 10122 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:21.041075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:54511 (local address 127.0.202.190:37521)
0927 21:40:21.041250 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:21.041256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:21.041566 (+ 310us) server_negotiation.cc:407] Connection header received
0927 21:40:21.041910 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:21.041915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:21.042004 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:21.042103 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:21.043017 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.043911 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.046025 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.046576 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.046675 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:21.057750 (+ 11075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:54511: BlockingRecv error: recv got EOF from 127.0.202.130:54511 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 21:40:21 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:21 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:21.595306 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:21.585047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:48065)
0927 21:40:21.586007 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:21.586025 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:21.586198 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:21.586563 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:21.586568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:21.587055 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:21.587263 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:21.587273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.588512 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.588519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:21.589531 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:21.589543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.590329 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.590335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:21.590536 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:21.591224 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:21.591244 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:21.595073 (+ 3829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":299,"threads_started":1}
W20260927 21:40:21.595692 10126 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:21.585369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:48065 (local address 127.0.202.190:37521)
0927 21:40:21.585831 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:21.585836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:21.586118 (+ 282us) server_negotiation.cc:407] Connection header received
0927 21:40:21.586336 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:21.586348 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:21.586412 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:21.586530 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:21.587423 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.588355 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.589691 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.590156 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.590311 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:21.595306 (+ 4995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:48065: BlockingRecv error: recv got EOF from 127.0.202.129:48065 (error 108)
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":248,"threads_started":1}
Sep 27 21:40:21 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:21 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:21.896726 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:21.886048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:40469)
0927 21:40:21.886703 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:21.886728 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:21.886848 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:21.887285 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:21.887291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:21.887896 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:21.888063 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:21.888072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.889655 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.889661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:21.890816 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:21.890829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.891750 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.891755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:21.891891 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:21.892613 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:21.892631 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:21.896313 (+ 3682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":327,"threads_started":1}
W20260927 21:40:21.896813 10126 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:21.886239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:40469 (local address 127.0.202.190:37521)
0927 21:40:21.886423 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:21.886433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:21.886822 (+ 389us) server_negotiation.cc:407] Connection header received
0927 21:40:21.886974 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:21.886980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:21.887075 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:21.887174 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:21.888368 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.889483 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.890983 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:21.891594 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:21.891720 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:21.896620 (+ 4900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:40469: BlockingRecv error: recv got EOF from 127.0.202.131:40469 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 21:40:22 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:22 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:22.069542 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:22.058807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:48699)
0927 21:40:22.059323 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:22.059341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:22.059454 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:22.059798 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:22.059801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:22.060200 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:22.060388 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:22.060397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.061484 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.061488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:22.062558 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:22.062571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.063804 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.063810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:22.064011 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:22.065031 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:22.065067 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:22.069204 (+ 4137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":288,"threads_started":1}
W20260927 21:40:22.069664 10126 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:22.058963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:48699 (local address 127.0.202.190:37521)
0927 21:40:22.059147 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:22.059154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:22.059420 (+ 266us) server_negotiation.cc:407] Connection header received
0927 21:40:22.059558 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:22.059562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:22.059640 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:22.059738 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:22.060525 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.061344 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.062844 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.063540 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.063787 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:22.069395 (+ 5608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:48699: BlockingRecv error: recv got EOF from 127.0.202.130:48699 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 21:40:22 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:22 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:22.606215 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:22.596281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:55353)
0927 21:40:22.596947 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:22.596963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:22.597078 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:22.597400 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:22.597403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:22.597787 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:22.597947 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:22.597954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.599003 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.599007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:22.599973 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:22.599985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.600863 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.600868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:22.600988 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:22.601625 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:22.601643 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:22.605849 (+ 4206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":307,"threads_started":1}
W20260927 21:40:22.606491 10130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:22.596423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:55353 (local address 127.0.202.190:37521)
0927 21:40:22.596962 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:22.596967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:22.597117 (+ 150us) server_negotiation.cc:407] Connection header received
0927 21:40:22.597175 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:22.597179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:22.597262 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:22.597357 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:22.598109 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.598856 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.600164 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.600665 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.600780 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:22.606140 (+ 5360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:55353: BlockingRecv error: recv got EOF from 127.0.202.129:55353 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":337,"threads_started":1}
Sep 27 21:40:22 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:22 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:22.907073 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:22.897813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:38865)
0927 21:40:22.898379 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:22.898397 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:22.898515 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:22.898926 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:22.898930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:22.899377 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:22.899548 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:22.899557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.900870 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.900875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:22.901829 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:22.901842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.902517 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.902521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:22.902646 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:22.903308 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:22.903326 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:22.906846 (+ 3520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":275,"threads_started":1}
W20260927 21:40:22.907315 10130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:22.897950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:38865 (local address 127.0.202.190:37521)
0927 21:40:22.898116 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:22.898121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:22.898473 (+ 352us) server_negotiation.cc:407] Connection header received
0927 21:40:22.898679 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:22.898684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:22.898754 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:22.898851 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:22.899713 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.900712 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.901981 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:22.902369 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:22.902471 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:22.907071 (+ 4600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:38865: BlockingRecv error: recv got EOF from 127.0.202.131:38865 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 21:40:23 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:23 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:23.080639 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:23.070631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:42739)
0927 21:40:23.071223 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:23.071241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:23.071364 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:23.071773 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:23.071777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:23.072198 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:23.072433 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:23.072442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.073468 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.073473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:23.074522 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:23.074534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.075270 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.075275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:23.075402 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:23.076094 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:23.076112 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:23.080411 (+ 4299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":279,"threads_started":1}
W20260927 21:40:23.080853 10130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:23.070669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:42739 (local address 127.0.202.190:37521)
0927 21:40:23.070939 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:23.070949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:23.071347 (+ 398us) server_negotiation.cc:407] Connection header received
0927 21:40:23.071470 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:23.071474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:23.071550 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:23.071633 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:23.072565 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.073326 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.074671 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.075104 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.075205 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:23.080576 (+ 5371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:42739: BlockingRecv error: recv got EOF from 127.0.202.130:42739 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 27 21:40:23 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:23 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:23.617729 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:23.607456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:52719)
0927 21:40:23.608087 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:23.608105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:23.608219 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:23.608781 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:23.608785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:23.609215 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:23.609375 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:23.609383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.611047 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.611058 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:23.611992 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:23.612003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.612962 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.612968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:23.613104 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:23.613721 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:23.613739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:23.617478 (+ 3739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":383,"threads_started":1}
W20260927 21:40:23.618089 10134 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:23.607535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:52719 (local address 127.0.202.190:37521)
0927 21:40:23.608487 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:23.608492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:23.608508 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:40:23.608553 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:23.608556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:23.608620 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:23.608709 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:23.609512 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.610756 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.612126 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.612587 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.612701 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:23.617814 (+ 5113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:52719: BlockingRecv error: recv got EOF from 127.0.202.129:52719 (error 108)
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":754,"threads_started":1}
Sep 27 21:40:23 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:23 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:23.917467 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:23.908024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:60279)
0927 21:40:23.908702 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:23.908723 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:23.908892 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:23.909254 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:23.909258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:23.909660 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:23.909827 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:23.909836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.910860 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.910867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:23.911846 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:23.911858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.912642 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.912648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:23.912864 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:23.913480 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:23.913496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:23.917115 (+ 3619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":329,"threads_started":1}
W20260927 21:40:23.917707 10134 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:23.908205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:60279 (local address 127.0.202.190:37521)
0927 21:40:23.908497 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:23.908506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:23.908819 (+ 313us) server_negotiation.cc:407] Connection header received
0927 21:40:23.909018 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:23.909025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:23.909092 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:23.909200 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:23.909975 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.910726 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.911989 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:23.912486 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:23.912626 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:23.917442 (+ 4816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:60279: BlockingRecv error: recv got EOF from 127.0.202.131:60279 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 27 21:40:24 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:24 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:24.091594 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:24.081714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:45429)
0927 21:40:24.082665 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:24.082685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:24.082826 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:24.083173 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:24.083178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:24.083605 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:24.083763 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:24.083772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.085027 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.085033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:24.085979 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:24.085989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.086714 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.086720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:24.086892 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:24.087557 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:24.087581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:24.091339 (+ 3758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":635,"threads_started":1}
W20260927 21:40:24.091949 10134 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:24.081802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:45429 (local address 127.0.202.190:37521)
0927 21:40:24.081987 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:24.081995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:24.082762 (+ 767us) server_negotiation.cc:407] Connection header received
0927 21:40:24.082956 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:24.082961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:24.083043 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:24.083145 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:24.083908 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.084888 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.086118 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.086530 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.086675 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:24.091641 (+ 4966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:45429: BlockingRecv error: recv got EOF from 127.0.202.130:45429 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 21:40:24 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:24 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:24.628383 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:24.618815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:33413)
0927 21:40:24.619344 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:24.619360 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:24.619486 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:24.619906 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:24.619910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:24.620405 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:24.620551 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:24.620559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.621633 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.621637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:24.622609 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:24.622620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.623440 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.623445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:24.623571 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:24.624162 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:24.624180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:24.628009 (+ 3829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":282,"threads_started":1}
W20260927 21:40:24.628530 10138 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:24.618852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:33413 (local address 127.0.202.190:37521)
0927 21:40:24.619364 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:24.619369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:24.619435 (+ 66us) server_negotiation.cc:407] Connection header received
0927 21:40:24.619654 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:24.619659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:24.619751 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:24.619849 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:24.620712 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.621491 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.622769 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.623274 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.623374 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:24.628219 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:33413: BlockingRecv error: recv got EOF from 127.0.202.129:33413 (error 108)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":277,"threads_started":1}
Sep 27 21:40:24 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:24 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:24.927664 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:24.918582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:47981)
0927 21:40:24.919144 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:24.919160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:24.919308 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:24.919668 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:24.919671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:24.920097 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:24.920269 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:24.920278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.921418 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.921423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:24.922397 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:24.922409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.923157 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.923162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:24.923274 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:24.923863 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:24.923881 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:24.927258 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":282,"threads_started":1}
W20260927 21:40:24.927747 10138 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:24.918665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:47981 (local address 127.0.202.190:37521)
0927 21:40:24.918848 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:24.918854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:24.919220 (+ 366us) server_negotiation.cc:407] Connection header received
0927 21:40:24.919441 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:24.919445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:24.919506 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:24.919586 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:24.920470 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.921268 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.922525 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:24.922973 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:24.923074 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:24.927509 (+ 4435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:47981: BlockingRecv error: recv got EOF from 127.0.202.131:47981 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 21:40:25 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:25 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:25.103575 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:25.092723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:41791)
0927 21:40:25.093416 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:25.093435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:25.093651 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:25.094229 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:25.094233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:25.094645 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:25.094810 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:25.094822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.095993 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.095997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:25.097236 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:25.097256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.098230 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.098236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:25.098391 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:25.099105 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:25.099127 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:25.103275 (+ 4148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":379,"threads_started":1}
W20260927 21:40:25.103777 10138 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:25.092757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:41791 (local address 127.0.202.190:37521)
0927 21:40:25.092950 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:25.092954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:25.093557 (+ 603us) server_negotiation.cc:407] Connection header received
0927 21:40:25.093882 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:25.093886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:25.093965 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:25.094054 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:25.095000 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.095800 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.097399 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.098051 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.098161 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:25.103515 (+ 5354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:41791: BlockingRecv error: recv got EOF from 127.0.202.130:41791 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 21:40:25 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:25 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:25.639709 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:25.629490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:54521)
0927 21:40:25.630137 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:25.630154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:25.630280 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:25.630660 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:25.630664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:25.631067 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:25.631256 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:25.631267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.632909 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.632914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:25.633984 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:25.633998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.634969 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.634974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:25.635100 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:25.635768 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:25.635788 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:25.639444 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":400,"threads_started":1}
W20260927 21:40:25.640318 10142 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:25.629521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:54521 (local address 127.0.202.190:37521)
0927 21:40:25.630139 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:25.630146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:25.630243 (+ 97us) server_negotiation.cc:407] Connection header received
0927 21:40:25.630402 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:25.630408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:25.630499 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:25.630636 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:25.631421 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.632696 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.634135 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.634642 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.634749 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:25.639714 (+ 4965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:54521: BlockingRecv error: recv got EOF from 127.0.202.129:54521 (error 108)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":381,"threads_started":1}
Sep 27 21:40:25 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:25 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:25.938030 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:25.929011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:49661)
0927 21:40:25.929655 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:25.929671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:25.929776 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:25.930174 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:25.930177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:25.930573 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:25.930782 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:25.930792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.931899 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.931903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:25.932941 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:25.932952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.933615 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.933620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:25.933743 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:25.934420 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:25.934438 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:25.937788 (+ 3350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":401,"threads_started":1}
W20260927 21:40:25.938337 10142 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:25.929103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:49661 (local address 127.0.202.190:37521)
0927 21:40:25.929328 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:25.929335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:25.929744 (+ 409us) server_negotiation.cc:407] Connection header received
0927 21:40:25.929905 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:25.929912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:25.930005 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:25.930129 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:25.930930 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.931733 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.933091 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:25.933477 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:25.933577 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:25.938031 (+ 4454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:49661: BlockingRecv error: recv got EOF from 127.0.202.131:49661 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 21:40:26 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:26 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:26.123745 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:26.104897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:57047)
0927 21:40:26.105549 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:26.105568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:26.105814 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:26.106236 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:26.106242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:26.106734 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:26.106911 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:26.106920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.108057 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.108065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:26.109245 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:26.109258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.110054 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.110061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:26.110254 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:26.119272 (+ 9018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:26.119297 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:26.123480 (+ 4183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":350,"threads_started":1}
W20260927 21:40:26.123945 10142 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:26.104990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:57047 (local address 127.0.202.190:37521)
0927 21:40:26.105249 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:26.105254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:26.105668 (+ 414us) server_negotiation.cc:407] Connection header received
0927 21:40:26.105967 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:26.105972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:26.106072 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:26.106196 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:26.107074 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.107894 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.109399 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.109874 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.110034 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:26.123682 (+ 13648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:57047: BlockingRecv error: recv got EOF from 127.0.202.130:57047 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 27 21:40:26 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:26 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:26.650684 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:26.640823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:52005)
0927 21:40:26.641291 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:26.641307 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:26.641421 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:26.641964 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:26.641969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:26.642482 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:26.642661 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:26.642671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.643909 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.643915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:26.645009 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:26.645022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.645883 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.645890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:26.646066 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:26.646849 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:26.646868 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:26.650365 (+ 3497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
W20260927 21:40:26.650870 10146 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:26.641126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:52005 (local address 127.0.202.190:37521)
0927 21:40:26.641611 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:26.641616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:26.641632 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:40:26.641682 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:26.641686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:26.641751 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:26.641844 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:26.642894 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.643773 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.645156 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.645721 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.645851 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:26.650590 (+ 4739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:52005: BlockingRecv error: recv got EOF from 127.0.202.129:52005 (error 108)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":282,"threads_started":1}
Sep 27 21:40:26 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:26 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:26.950937 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:26.939361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:46481)
0927 21:40:26.940049 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:26.940071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:26.940216 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:26.940785 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:26.940788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:26.941274 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:26.941508 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:26.941524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.942886 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.942889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:26.944106 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:26.944123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.944894 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.944899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:26.945058 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:26.945854 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:26.945880 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:26.950619 (+ 4739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":347,"threads_started":1}
W20260927 21:40:26.951247 10146 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:26.939767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:46481 (local address 127.0.202.190:37521)
0927 21:40:26.940060 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:26.940065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:26.940138 (+ 73us) server_negotiation.cc:407] Connection header received
0927 21:40:26.940382 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:26.940388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:26.940488 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:26.940583 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:26.941682 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.942736 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.944307 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:26.944729 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:26.944879 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:26.950955 (+ 6076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:46481: BlockingRecv error: recv got EOF from 127.0.202.131:46481 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 27 21:40:27 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:27 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:27.134279 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:27.124958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:37173)
0927 21:40:27.125465 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:27.125485 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:27.125616 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:27.125998 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:27.126003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:27.126420 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:27.126585 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:27.126594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.127649 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.127655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:27.128622 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:27.128634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.129329 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.129335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:27.129511 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:27.130085 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:27.130104 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:27.133955 (+ 3851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":341,"threads_started":1}
W20260927 21:40:27.134404 10146 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:27.125101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:37173 (local address 127.0.202.190:37521)
0927 21:40:27.125380 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:27.125386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:27.125595 (+ 209us) server_negotiation.cc:407] Connection header received
0927 21:40:27.125758 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:27.125766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:27.125854 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:27.125963 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:27.126746 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.127513 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.128764 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.129166 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.129295 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:27.134144 (+ 4849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:37173: BlockingRecv error: recv got EOF from 127.0.202.130:37173 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 21:40:27 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:27 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:27.662356 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:27.651898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:58535)
0927 21:40:27.652593 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:27.652616 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:27.652763 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:27.653189 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:27.653194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:27.653651 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:27.653858 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:27.653875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.655437 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.655441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:27.656672 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:27.656692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.657450 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.657460 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:27.657583 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:27.658234 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:27.658251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:27.662058 (+ 3807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":280,"threads_started":1}
W20260927 21:40:27.662488 10150 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:27.651990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:58535 (local address 127.0.202.190:37521)
0927 21:40:27.652711 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:27.652717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:27.652750 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:40:27.652933 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:27.652940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:27.653018 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:27.653140 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:27.654026 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.655277 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.656840 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.657290 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.657392 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:27.662280 (+ 4888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:58535: BlockingRecv error: recv got EOF from 127.0.202.129:58535 (error 108)
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":493,"threads_started":1}
Sep 27 21:40:27 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:27 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:27.961490 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:27.952051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:48301)
0927 21:40:27.952608 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:27.952624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:27.952782 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:27.953110 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:27.953115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:27.953578 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:27.953803 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:27.953815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.955343 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.955348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:27.956368 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:27.956384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.957076 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.957080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:27.957195 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:27.957861 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:27.957884 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:27.961278 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":324,"threads_started":1}
W20260927 21:40:27.961783 10150 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:27.952150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:48301 (local address 127.0.202.190:37521)
0927 21:40:27.952424 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:27.952432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:27.952707 (+ 275us) server_negotiation.cc:407] Connection header received
0927 21:40:27.952912 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:27.952923 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:27.952981 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:27.953082 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:27.954026 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.955199 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.956504 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:27.956942 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:27.957048 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:27.961498 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:48301: BlockingRecv error: recv got EOF from 127.0.202.131:48301 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 21:40:28 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:28 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:28.145501 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:28.135445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:35863)
0927 21:40:28.135994 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:28.136010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:28.136134 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:28.136510 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:28.136514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:28.136925 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:28.137095 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:28.137104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.138194 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.138207 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:28.139345 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:28.139358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.140670 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.140680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:28.140895 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:28.141652 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:28.141670 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:28.145202 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":314,"threads_started":1}
W20260927 21:40:28.145712 10150 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:28.135481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:35863 (local address 127.0.202.190:37521)
0927 21:40:28.135671 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:28.135676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:28.136105 (+ 429us) server_negotiation.cc:407] Connection header received
0927 21:40:28.136253 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:28.136256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:28.136352 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:28.136436 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:28.137260 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.138023 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.139523 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.140413 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.140599 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:28.145434 (+ 4835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:35863: BlockingRecv error: recv got EOF from 127.0.202.130:35863 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 21:40:28 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:28 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:28.671900 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:28.663340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:48421)
0927 21:40:28.663832 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:28.663848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:28.663959 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:28.664251 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:28.664254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:28.664695 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:28.664850 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:28.664882 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.665867 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.665871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:28.666755 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:28.666766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.667458 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.667462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:28.667579 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:28.668236 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:28.668253 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:28.671669 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":261,"threads_started":1}
W20260927 21:40:28.672194 10154 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:28.663347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:48421 (local address 127.0.202.190:37521)
0927 21:40:28.663845 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:28.663850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:28.663921 (+ 71us) server_negotiation.cc:407] Connection header received
0927 21:40:28.664050 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:28.664054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:28.664124 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:28.664212 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:28.665010 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.665737 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.666894 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.667324 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.667417 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:28.671924 (+ 4507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:48421: BlockingRecv error: recv got EOF from 127.0.202.129:48421 (error 108)
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":272,"threads_started":1}
Sep 27 21:40:28 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:28 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:28.971760 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:28.962472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:55071)
0927 21:40:28.962995 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:28.963013 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:28.963136 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:28.963461 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:28.963465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:28.963965 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:28.964156 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:28.964165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.965202 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.965206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:28.966190 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:28.966202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.966961 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.966966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:28.967094 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:28.967679 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:28.967697 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:28.971516 (+ 3819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":274,"threads_started":1}
W20260927 21:40:28.971971 10154 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:28.962588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:55071 (local address 127.0.202.190:37521)
0927 21:40:28.962833 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:28.962839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:28.963098 (+ 259us) server_negotiation.cc:407] Connection header received
0927 21:40:28.963246 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:28.963249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:28.963309 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:28.963385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:28.964315 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.965044 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.966330 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:28.966761 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:28.966860 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:28.971718 (+ 4858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:55071: BlockingRecv error: recv got EOF from 127.0.202.131:55071 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 21:40:29 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:29 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:29.156563 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:29.146708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:57345)
0927 21:40:29.147237 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:29.147275 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:29.147448 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:29.147814 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:29.147817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:29.148259 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:29.148426 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:29.148446 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.149613 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.149617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:29.150504 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:29.150514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.151464 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.151469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:29.151597 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:29.152371 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:29.152388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:29.156281 (+ 3893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
W20260927 21:40:29.156908 10154 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:29.146944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:57345 (local address 127.0.202.190:37521)
0927 21:40:29.147124 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:29.147130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:29.147422 (+ 292us) server_negotiation.cc:407] Connection header received
0927 21:40:29.147573 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:29.147577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:29.147673 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:29.147782 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:29.148590 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.149482 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.150668 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.151251 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.151401 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:29.156587 (+ 5186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:57345: BlockingRecv error: recv got EOF from 127.0.202.130:57345 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 21:40:29 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:29 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.129@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:29.683015 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:29.673130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.129:45731)
0927 21:40:29.673652 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:29.673669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:29.673769 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:29.674105 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:29.674109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:29.674483 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:29.674663 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:29.674672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.675724 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.675727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:29.676763 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:29.676775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.677576 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.677580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:29.677779 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:29.678510 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:29.678526 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:29.682623 (+ 4097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
W20260927 21:40:29.683072 10158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:29.673265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:45731 (local address 127.0.202.190:37521)
0927 21:40:29.673727 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:29.673732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:29.673752 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:40:29.673877 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:29.673881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:29.673958 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:29.674054 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:29.674827 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.675595 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.676919 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.677427 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.677570 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:29.682836 (+ 5266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.129:45731: BlockingRecv error: recv got EOF from 127.0.202.129:45731 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":266,"threads_started":1}
Sep 27 21:40:29 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:29 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.131@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:29.981873 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:29.973011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.131:57115)
0927 21:40:29.973664 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:29.973679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:29.973839 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:29.974176 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:29.974181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:29.974592 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:29.974730 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:29.974738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.975745 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.975750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:29.976681 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:29.976693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.977382 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.977388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:29.977560 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:29.978152 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:29.978169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:29.981540 (+ 3371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":252,"threads_started":1}
W20260927 21:40:29.981997 10158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:29.973044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:57115 (local address 127.0.202.190:37521)
0927 21:40:29.973395 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:29.973401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:29.973770 (+ 369us) server_negotiation.cc:407] Connection header received
0927 21:40:29.973961 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:29.973968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:29.974040 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:29.974144 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:29.974874 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.975598 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.976824 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:29.977229 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:29.977367 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:29.981737 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.131:57115: BlockingRecv error: recv got EOF from 127.0.202.131:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":205}
Sep 27 21:40:30 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:40:30 dist-test-slave-25dq krb5kdc[9098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.202.130@KRBTEST.COM for oryx/127.0.202.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:40:30.168123 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:30.158037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:37521 (local address 127.0.202.130:38063)
0927 21:40:30.158565 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:30.158581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:30.158757 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:30.159139 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:30.159143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:30.159643 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:30.159942 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:30.159954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.161231 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.161239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:30.162264 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:30.162278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.163035 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.163042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:30.163259 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:30.163940 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:30.163959 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:30.167749 (+ 3790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.202.190:37521: Server oryx/127.0.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
W20260927 21:40:30.168195 10158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:30.158174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:38063 (local address 127.0.202.190:37521)
0927 21:40:30.158370 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:30.158376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:30.158676 (+ 300us) server_negotiation.cc:407] Connection header received
0927 21:40:30.158940 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:30.158944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:30.159010 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:30.159113 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:30.160140 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.161071 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.162425 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.162864 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.163037 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:30.167953 (+ 4916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.202.130:38063: BlockingRecv error: recv got EOF from 127.0.202.130:38063 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 21:40:30.633441 9650 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.0.0.1:47412 (request call id 2390) took 2 ms (client timeout 2 ms). Trace:
W20260927 21:40:30.633641 9650 rpcz_store.cc:269] 0927 21:40:30.630737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 21:40:30.633186 (+ 2449us) service_pool.cc:214] Skipping call since client already timed out
0927 21:40:30.633394 (+ 208us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 21:40:30.644447 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9687
I20260927 21:40:30.657258 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:30.645661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.202.129:41457)
0927 21:40:30.646350 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:30.646357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:30.646382 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:40:30.646459 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:30.646465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:30.646698 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:30.646867 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:30.648214 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.649472 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.650686 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.651195 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.651291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:30.653935 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:30.653945 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:30.653961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:30.653996 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:30.656760 (+ 2764us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.202.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:40:30.657037 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33548: Request ticket server kudu/127.0.202.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":375,"threads_started":1}
W20260927 21:40:30.657378 10162 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33548: Request ticket server kudu/127.0.202.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 21:40:30.657521 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9687: Not authorized: Client connection negotiation failed: client connection to 127.0.202.129:41457: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.202.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:40:30.657757 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9687
I20260927 21:40:30.675254 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9838
I20260927 21:40:30.688818 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:30.676343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.0.202.130:33265)
0927 21:40:30.676880 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:30.676887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:30.676912 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:40:30.676979 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:30.676985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:30.677213 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:30.677381 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:30.679253 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.680259 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.681677 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.682219 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.682322 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:30.685230 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:30.685238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:30.685252 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:30.685294 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:30.688144 (+ 2850us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.202.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:40:30.688566 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37644: Request ticket server kudu/127.0.202.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":273,"threads_started":1}
W20260927 21:40:30.688961 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9838: Not authorized: Client connection negotiation failed: client connection to 127.0.202.130:33265: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.202.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:40:30.689090 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9838
I20260927 21:40:30.706310 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9971
I20260927 21:40:30.720520 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:30.707422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.202.131:37905)
0927 21:40:30.707924 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:30.707932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:30.707958 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:40:30.708018 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:30.708025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:30.708267 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:30.708562 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:30.710328 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.711410 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.712932 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:30.713619 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:30.713779 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:30.716728 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:30.716738 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:30.716754 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:30.716820 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:30.719609 (+ 2789us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.202.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:40:30.720117 (+ 508us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52600: Request ticket server kudu/127.0.202.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":224,"threads_started":1}
W20260927 21:40:30.720496 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9971: Not authorized: Client connection negotiation failed: client connection to 127.0.202.131:37905: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.202.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 21:40:30.720606 10164 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52600: Request ticket server kudu/127.0.202.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:40:30.720701 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9971
I20260927 21:40:30.737108 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 9618
I20260927 21:40:30.738538 9680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:30.856731 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 9618
2026-09-27T21:40:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21879 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T21:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:40:30Z Disabled control of system clock
I20260927 21:40:30.934547 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40273
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:40:31.284852 10169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:31.285290 10169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:31.285399 10169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:31.292933 10169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:40:31.293053 10169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:31.293092 10169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:31.293139 10169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:40:31.293182 10169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:40:31.302362 10169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:31.303799 10169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:31.305536 10169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:31.317605 10177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:31.317672 10174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:31.317684 10175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:31.318488 10169 server_base.cc:1034] running on GCE node
I20260927 21:40:31.319278 10169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:31.320756 10169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:31.321951 10169 hybrid_clock.cc:648] HybridClock initialized: now 1790545231321891 us; error 55 us; skew 500 ppm
I20260927 21:40:31.322260 10169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:31.329435 10169 webserver.cc:460] Webserver started at http://127.0.202.190:35211/ using document root <none> and password file <none>
I20260927 21:40:31.330183 10169 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:31.330273 10169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:31.330547 10169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:31.333884 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8"
format_stamp: "Formatted at 2026-09-27 21:40:31 on dist-test-slave-25dq"
server_key: "a3f962cf530a2fa07d3bc79b08243ad6"
server_key_iv: "2a19f5f13e27d29955c959d232b2849a"
server_key_version: "encryptionkey@0"
I20260927 21:40:31.334676 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8"
format_stamp: "Formatted at 2026-09-27 21:40:31 on dist-test-slave-25dq"
server_key: "a3f962cf530a2fa07d3bc79b08243ad6"
server_key_iv: "2a19f5f13e27d29955c959d232b2849a"
server_key_version: "encryptionkey@0"
I20260927 21:40:31.340080 10169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:40:31.343536 10183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:31.345036 10169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:31.345221 10169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8"
format_stamp: "Formatted at 2026-09-27 21:40:31 on dist-test-slave-25dq"
server_key: "a3f962cf530a2fa07d3bc79b08243ad6"
server_key_iv: "2a19f5f13e27d29955c959d232b2849a"
server_key_version: "encryptionkey@0"
I20260927 21:40:31.345459 10169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:31.375052 10169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:31.376039 10169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:31.376398 10169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:31.395891 10169 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40273
I20260927 21:40:31.396005 10234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40273 every 8 connection(s)
I20260927 21:40:31.397418 10169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:40:31.398064 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 10169
I20260927 21:40:31.398324 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:40:31.399116 810 external_mini_cluster.cc:1428] Setting key 89d348e57920058a5711edb1220e10fc
I20260927 21:40:31.401790 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:31.415690 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:31.401422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.202.190:40273)
0927 21:40:31.402616 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:31.402630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:31.402822 (+ 192us) server_negotiation.cc:407] Connection header received
0927 21:40:31.404939 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:31.404969 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:31.405646 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:31.406132 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:31.406678 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:31.408720 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:31.410290 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:31.411275 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:31.411463 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:31.412770 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:31.412781 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:31.412805 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:31.413556 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:31.413664 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:31.413693 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:31.413820 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:31.414079 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:40:31.414410 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":408,"threads_started":1}
I20260927 21:40:31.416242 10235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8: Bootstrap starting.
I20260927 21:40:31.419559 10235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:31.420765 10235 log.cc:824] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:31.423833 10235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8: No bootstrap required, opened a new log
I20260927 21:40:31.431571 10235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40273 } }
I20260927 21:40:31.431967 10235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:31.432062 10235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9d7aeea8b842cf9ed22a6cc0a1b9b8, State: Initialized, Role: FOLLOWER
I20260927 21:40:31.432696 10235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40273 } }
I20260927 21:40:31.432924 10235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:40:31.433055 10235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:40:31.433254 10235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:31.435411 10235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40273 } }
I20260927 21:40:31.435925 10235 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9d7aeea8b842cf9ed22a6cc0a1b9b8; no voters:
I20260927 21:40:31.436595 10235 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:40:31.436774 10240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:31.437597 10240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [term 1 LEADER]: Becoming Leader. State: Replica: 9a9d7aeea8b842cf9ed22a6cc0a1b9b8, State: Running, Role: LEADER
I20260927 21:40:31.438292 10240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40273 } }
I20260927 21:40:31.439105 10235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:40:31.441740 10242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9d7aeea8b842cf9ed22a6cc0a1b9b8. Latest consensus state: current_term: 1 leader_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40273 } } }
I20260927 21:40:31.442250 10242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:31.441766 10241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9d7aeea8b842cf9ed22a6cc0a1b9b8" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40273 } } }
I20260927 21:40:31.442378 10241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:31.444381 10248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:40:31.449620 10248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:40:31.458956 10248 catalog_manager.cc:1347] Generated new cluster ID: a5a210be0c034e03bb169bc2b6f7c32b
I20260927 21:40:31.459100 10248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:40:31.470216 10248 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:40:31.471624 10248 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:40:31.479152 10248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a9d7aeea8b842cf9ed22a6cc0a1b9b8: Generated new TSK 0
I20260927 21:40:31.480115 10248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:40:31.491652 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40273
--builtin_ntp_servers=127.0.202.148:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:40:31.852114 10259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:31.852496 10259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:31.852586 10259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:31.860061 10259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:31.860175 10259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:31.860355 10259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:40:31.869673 10259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:31.871141 10259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:31.873065 10259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:31.887518 10265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:31.887974 10264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:31.887970 10267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:31.888612 10259 server_base.cc:1034] running on GCE node
I20260927 21:40:31.889240 10259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:31.890290 10259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:31.891599 10259 hybrid_clock.cc:648] HybridClock initialized: now 1790545231891548 us; error 81 us; skew 500 ppm
I20260927 21:40:31.891978 10259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:31.894646 10259 webserver.cc:460] Webserver started at http://127.0.202.129:44757/ using document root <none> and password file <none>
I20260927 21:40:31.895411 10259 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:31.895548 10259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:31.895918 10259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:31.899569 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "0683ab76f5e640a98fd29faf8e8897af"
format_stamp: "Formatted at 2026-09-27 21:40:31 on dist-test-slave-25dq"
server_key: "6bf3771a9efefe1bb35ba9e7d1e7cfcc"
server_key_iv: "08ca3db7b618f28c787836a575a641a1"
server_key_version: "encryptionkey@0"
I20260927 21:40:31.900640 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "0683ab76f5e640a98fd29faf8e8897af"
format_stamp: "Formatted at 2026-09-27 21:40:31 on dist-test-slave-25dq"
server_key: "6bf3771a9efefe1bb35ba9e7d1e7cfcc"
server_key_iv: "08ca3db7b618f28c787836a575a641a1"
server_key_version: "encryptionkey@0"
I20260927 21:40:31.906721 10259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:40:31.911015 10273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:31.912758 10259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:31.912983 10259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "0683ab76f5e640a98fd29faf8e8897af"
format_stamp: "Formatted at 2026-09-27 21:40:31 on dist-test-slave-25dq"
server_key: "6bf3771a9efefe1bb35ba9e7d1e7cfcc"
server_key_iv: "08ca3db7b618f28c787836a575a641a1"
server_key_version: "encryptionkey@0"
I20260927 21:40:31.913228 10259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:31.932201 10259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:31.933219 10259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:31.933566 10259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:31.934600 10259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:31.936219 10259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:31.936378 10259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:31.936522 10259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:31.936619 10259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:31.975250 10259 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:39849
I20260927 21:40:31.975288 10385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:39849 every 8 connection(s)
I20260927 21:40:31.976786 10259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:40:31.980051 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 10259
I20260927 21:40:31.980257 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:40:31.981245 810 external_mini_cluster.cc:1428] Setting key 41d95d30b4d4d431997183cdfbcde5e6
I20260927 21:40:31.984442 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40273
--builtin_ntp_servers=127.0.202.148:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:40:31.991799 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:31.979568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:40283 (local address 127.0.202.190:40273)
0927 21:40:31.980034 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:31.980039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:31.981686 (+ 1647us) server_negotiation.cc:407] Connection header received
0927 21:40:31.983600 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:31.983605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:31.983667 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:31.983761 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:31.985271 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:31.986058 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:31.988546 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:31.989115 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:31.989224 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:31.991003 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:31.991008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:31.991011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:31.991029 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:31.991067 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:31.991079 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:31.991151 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:31.991432 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 21:40:31.991509 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":259,"threads_started":1}
I20260927 21:40:31.993085 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:31.980409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40273 (local address 127.0.202.129:40283)
0927 21:40:31.981528 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:31.981573 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:31.983282 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:31.984073 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:31.984088 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:31.984209 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:31.985083 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:31.985112 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:31.986220 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:31.986228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:31.988365 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:31.988382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:31.989349 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:31.989359 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:31.989691 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:31.990546 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:40:31.990552 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:31.990607 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:40:31.990720 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:31.991179 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:31.991193 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:31.991204 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:40:31.991388 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 21:40:31.991841 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":348,"threads_started":1}
I20260927 21:40:31.994881 10386 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40273
I20260927 21:40:31.995200 10386 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:31.996022 10386 heartbeater.cc:511] Master 127.0.202.190:40273 requested a full tablet report, sending...
I20260927 21:40:31.998333 10200 ts_manager.cc:194] Registered new tserver with Master: 0683ab76f5e640a98fd29faf8e8897af (127.0.202.129:39849)
I20260927 21:40:32.000622 10200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.129:40283
W20260927 21:40:32.324093 10390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:32.324472 10390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:32.324563 10390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:32.331812 10390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:32.331916 10390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:32.332021 10390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:40:32.341476 10390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:32.342756 10390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:32.344496 10390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:32.358384 10398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:32.358449 10395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:32.358652 10396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:32.359511 10390 server_base.cc:1034] running on GCE node
I20260927 21:40:32.360231 10390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:32.361284 10390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:32.362483 10390 hybrid_clock.cc:648] HybridClock initialized: now 1790545232362431 us; error 41 us; skew 500 ppm
I20260927 21:40:32.362814 10390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:32.365283 10390 webserver.cc:460] Webserver started at http://127.0.202.130:34733/ using document root <none> and password file <none>
I20260927 21:40:32.366065 10390 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:32.366156 10390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:32.366489 10390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:32.369494 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "ae701b37ab7240d983c8c26734b4e1d9"
format_stamp: "Formatted at 2026-09-27 21:40:32 on dist-test-slave-25dq"
server_key: "caac4cddadf2ec55aa8f5d97942d6085"
server_key_iv: "6584d73b02300423e61bb7e4d64bf9bb"
server_key_version: "encryptionkey@0"
I20260927 21:40:32.370401 10390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "ae701b37ab7240d983c8c26734b4e1d9"
format_stamp: "Formatted at 2026-09-27 21:40:32 on dist-test-slave-25dq"
server_key: "caac4cddadf2ec55aa8f5d97942d6085"
server_key_iv: "6584d73b02300423e61bb7e4d64bf9bb"
server_key_version: "encryptionkey@0"
I20260927 21:40:32.375706 10390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:40:32.379511 10404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:32.381057 10390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:32.381237 10390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "ae701b37ab7240d983c8c26734b4e1d9"
format_stamp: "Formatted at 2026-09-27 21:40:32 on dist-test-slave-25dq"
server_key: "caac4cddadf2ec55aa8f5d97942d6085"
server_key_iv: "6584d73b02300423e61bb7e4d64bf9bb"
server_key_version: "encryptionkey@0"
I20260927 21:40:32.381457 10390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:32.392762 10390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:32.393571 10390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:32.393872 10390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:32.394812 10390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:32.396402 10390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:32.396516 10390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:32.396646 10390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:32.396733 10390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:32.434016 10390 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:37465
I20260927 21:40:32.434106 10516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:37465 every 8 connection(s)
I20260927 21:40:32.435356 10390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:40:32.441883 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 10390
I20260927 21:40:32.442070 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:40:32.442664 810 external_mini_cluster.cc:1428] Setting key e08666f787d8c67f80a577bdbe074aaf
I20260927 21:40:32.445792 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40273
--builtin_ntp_servers=127.0.202.148:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:40:32.449219 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:32.438196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:53519 (local address 127.0.202.190:40273)
0927 21:40:32.438392 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:32.438397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:32.439913 (+ 1516us) server_negotiation.cc:407] Connection header received
0927 21:40:32.442232 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:32.442237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:32.442306 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:32.442392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:32.443796 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.444738 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.446431 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.446835 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.446981 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:32.448444 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:32.448449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:32.448453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:32.448470 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:32.448500 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:32.448525 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:32.448636 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:32.448921 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 21:40:32.449002 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:40:32.451093 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:32.438659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40273 (local address 127.0.202.130:53519)
0927 21:40:32.439745 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:32.439783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:32.441890 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:32.442682 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:32.442698 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:32.442857 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:32.443603 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:32.443623 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.444874 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.444881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:32.446292 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:32.446303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.447062 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.447069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:32.447292 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:32.448027 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:40:32.448029 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:32.448070 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:40:32.448154 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:32.448653 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:32.448668 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:32.448679 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:40:32.448886 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 21:40:32.449476 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":367,"threads_started":1}
I20260927 21:40:32.452739 10517 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40273
I20260927 21:40:32.453078 10517 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:32.453837 10517 heartbeater.cc:511] Master 127.0.202.190:40273 requested a full tablet report, sending...
I20260927 21:40:32.455360 10200 ts_manager.cc:194] Registered new tserver with Master: ae701b37ab7240d983c8c26734b4e1d9 (127.0.202.130:37465)
I20260927 21:40:32.456645 10200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.130:53519
W20260927 21:40:32.800171 10520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:32.800545 10520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:32.800637 10520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:32.807890 10520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:32.808012 10520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:32.808117 10520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:40:32.817567 10520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:32.818913 10520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:32.820636 10520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:32.834594 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:32.834522 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:32.834560 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:32.835331 10520 server_base.cc:1034] running on GCE node
I20260927 21:40:32.835907 10520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:32.836920 10520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:32.838177 10520 hybrid_clock.cc:648] HybridClock initialized: now 1790545232838090 us; error 47 us; skew 500 ppm
I20260927 21:40:32.838487 10520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:32.841156 10520 webserver.cc:460] Webserver started at http://127.0.202.131:43949/ using document root <none> and password file <none>
I20260927 21:40:32.841812 10520 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:32.841895 10520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:32.842160 10520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:32.845116 10520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "89702faeed7949b485bd5cd829d36198"
format_stamp: "Formatted at 2026-09-27 21:40:32 on dist-test-slave-25dq"
server_key: "b999ddf411ce65fdaeec65b4ddfe3b61"
server_key_iv: "cb29b18870c9a5a5d08638146b8c7d54"
server_key_version: "encryptionkey@0"
I20260927 21:40:32.845935 10520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "89702faeed7949b485bd5cd829d36198"
format_stamp: "Formatted at 2026-09-27 21:40:32 on dist-test-slave-25dq"
server_key: "b999ddf411ce65fdaeec65b4ddfe3b61"
server_key_iv: "cb29b18870c9a5a5d08638146b8c7d54"
server_key_version: "encryptionkey@0"
I20260927 21:40:32.851316 10520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:40:32.855015 10534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:32.856679 10520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:32.856827 10520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "89702faeed7949b485bd5cd829d36198"
format_stamp: "Formatted at 2026-09-27 21:40:32 on dist-test-slave-25dq"
server_key: "b999ddf411ce65fdaeec65b4ddfe3b61"
server_key_iv: "cb29b18870c9a5a5d08638146b8c7d54"
server_key_version: "encryptionkey@0"
I20260927 21:40:32.857093 10520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:32.871685 10520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:32.872611 10520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:32.872923 10520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:32.874074 10520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:32.875564 10520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:32.875648 10520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:32.875800 10520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:32.875903 10520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:32.910689 10520 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:36901
I20260927 21:40:32.910737 10646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:36901 every 8 connection(s)
I20260927 21:40:32.912031 10520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:40:32.921836 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 10520
I20260927 21:40:32.922022 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:40:32.922549 810 external_mini_cluster.cc:1428] Setting key 93b3f7de3be44fd784c64f9ef7d4114b
I20260927 21:40:32.925720 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:32.914706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:46463 (local address 127.0.202.190:40273)
0927 21:40:32.914875 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:32.914880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:32.916511 (+ 1631us) server_negotiation.cc:407] Connection header received
0927 21:40:32.918335 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:32.918341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:32.918436 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:32.918552 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:32.919621 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.920450 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.922678 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.923090 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.923196 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:32.924911 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:32.924916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:32.924919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:32.924935 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:32.924965 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:32.924976 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:32.925061 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:32.925295 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:40:32.925371 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:40:32.926815 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:32.915394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40273 (local address 127.0.202.131:46463)
0927 21:40:32.916354 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:32.916402 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:32.918053 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:32.918785 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:32.918794 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:32.918885 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:32.919455 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:32.919471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.920605 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.920609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:32.922493 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:32.922508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.923329 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.923335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:32.923597 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:32.924553 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:40:32.924556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:32.924597 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:40:32.924707 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:32.925079 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:32.925088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:32.925097 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:40:32.925284 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 21:40:32.925737 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":286,"threads_started":1}
I20260927 21:40:32.928470 10647 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40273
I20260927 21:40:32.928745 10647 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:32.929484 10647 heartbeater.cc:511] Master 127.0.202.190:40273 requested a full tablet report, sending...
I20260927 21:40:32.930858 10200 ts_manager.cc:194] Registered new tserver with Master: 89702faeed7949b485bd5cd829d36198 (127.0.202.131:36901)
I20260927 21:40:32.931619 10200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.131:46463
I20260927 21:40:32.938269 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:40:32.945390 10388 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:32.943342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.202.190:40273)
0927 21:40:32.943518 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:32.943523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:32.944125 (+ 602us) server_negotiation.cc:407] Connection header received
0927 21:40:32.944246 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:32.944250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:32.944426 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:32.944536 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:32.945086 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37684: BlockingRecv error: recv got EOF from 127.0.0.1:37684 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:40:32.946379 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 10259
I20260927 21:40:32.954650 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:32.947389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.202.129:39849)
0927 21:40:32.947962 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:32.947971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:32.948007 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:40:32.948088 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:32.948094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:32.948395 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:32.948559 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:32.948947 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.950160 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.951418 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:32.952069 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:32.952177 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:32.953244 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:32.953252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:32.953265 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:32.953903 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:32.953953 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:32.954015 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:32.954108 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:32.954250 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:40:32.954369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":303,"threads_started":1}
I20260927 21:40:32.955925 10381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:33.037688 10386 heartbeater.cc:503] Master 127.0.202.190:40273 was elected leader, sending a full tablet report...
I20260927 21:40:33.038867 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 10259
W20260927 21:40:33.055413 10186 connection.cc:535] server connection from 127.0.202.129:40283 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:40:33.055593 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 10390
I20260927 21:40:33.063635 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:33.056490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.0.202.130:37465)
0927 21:40:33.057048 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:33.057056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:33.057082 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:40:33.057139 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:33.057146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:33.057365 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:33.057543 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:33.057949 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:33.058921 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:33.060138 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:33.060904 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:33.061041 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:33.062185 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:33.062194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:33.062208 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:33.062808 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:33.062854 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:33.062874 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:33.063009 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:33.063161 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:40:33.063287 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
I20260927 21:40:33.064922 10512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:33.147089 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 10390
I20260927 21:40:33.165642 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 10520
I20260927 21:40:33.174705 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:33.166708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.202.131:36901)
0927 21:40:33.167413 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:33.167421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:33.167456 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:40:33.167646 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:33.167655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:33.167948 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:33.168094 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:33.168732 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:33.169882 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:33.171256 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:33.171867 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:33.171976 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:33.173281 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:33.173289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:33.173303 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:33.173918 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:33.173970 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:33.173990 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:33.174083 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:33.174259 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 21:40:33.174400 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":365,"threads_started":1}
I20260927 21:40:33.175977 10642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:33.259497 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 10520
I20260927 21:40:33.275885 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 10169
I20260927 21:40:33.277225 10230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:33.383031 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 10169
2026-09-27T21:40:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2520 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T21:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:40:33Z Disabled control of system clock
I20260927 21:40:33.456844 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:36451
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:40:33.792017 10666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:33.792390 10666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:33.792500 10666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:33.799860 10666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:40:33.799969 10666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:33.800006 10666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:33.800055 10666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:40:33.800098 10666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:40:33.809393 10666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:36451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:33.810649 10666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:33.812394 10666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:33.824873 10671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:33.825350 10672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:33.825402 10674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:33.826087 10666 server_base.cc:1034] running on GCE node
I20260927 21:40:33.826944 10666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:33.828646 10666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:33.829888 10666 hybrid_clock.cc:648] HybridClock initialized: now 1790545233829846 us; error 59 us; skew 500 ppm
I20260927 21:40:33.830330 10666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:33.832911 10666 webserver.cc:460] Webserver started at http://127.0.202.190:45135/ using document root <none> and password file <none>
I20260927 21:40:33.833695 10666 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:33.833839 10666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:33.834200 10666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:33.837376 10666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "0ebc5e3ac5544f73b426031e97ca3b4c"
format_stamp: "Formatted at 2026-09-27 21:40:33 on dist-test-slave-25dq"
server_key: "1eb2bd2e73b970fae77724f5a49cb146"
server_key_iv: "e224027bf6a4b2351e0734c9555e63f4"
server_key_version: "encryptionkey@0"
I20260927 21:40:33.838239 10666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "0ebc5e3ac5544f73b426031e97ca3b4c"
format_stamp: "Formatted at 2026-09-27 21:40:33 on dist-test-slave-25dq"
server_key: "1eb2bd2e73b970fae77724f5a49cb146"
server_key_iv: "e224027bf6a4b2351e0734c9555e63f4"
server_key_version: "encryptionkey@0"
I20260927 21:40:33.843782 10666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:40:33.847779 10680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:33.849428 10666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:33.849638 10666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "0ebc5e3ac5544f73b426031e97ca3b4c"
format_stamp: "Formatted at 2026-09-27 21:40:33 on dist-test-slave-25dq"
server_key: "1eb2bd2e73b970fae77724f5a49cb146"
server_key_iv: "e224027bf6a4b2351e0734c9555e63f4"
server_key_version: "encryptionkey@0"
I20260927 21:40:33.849874 10666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:33.856034 10666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:33.856894 10666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:33.857208 10666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:33.876091 10666 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:36451
I20260927 21:40:33.876163 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:36451 every 8 connection(s)
I20260927 21:40:33.877602 10666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:40:33.879576 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 10666
I20260927 21:40:33.879781 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:40:33.880481 810 external_mini_cluster.cc:1428] Setting key 3498970459935ad0cd5d0edf8eb69b6c
I20260927 21:40:33.882494 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:33.889384 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:33.882594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.202.190:36451)
0927 21:40:33.883635 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:33.883648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:33.883690 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:40:33.885322 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:33.885347 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:33.885807 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:33.886246 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:33.887409 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:33.887415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:33.887430 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:33.887443 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:33.887518 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:33.887536 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:33.887805 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:33.887951 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:40:33.888228 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":387,"threads_started":1}
I20260927 21:40:33.896620 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c: Bootstrap starting.
I20260927 21:40:33.899993 10732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:33.901436 10732 log.cc:824] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:33.904755 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c: No bootstrap required, opened a new log
I20260927 21:40:33.913087 10732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36451 } }
I20260927 21:40:33.913517 10732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:33.913627 10732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebc5e3ac5544f73b426031e97ca3b4c, State: Initialized, Role: FOLLOWER
I20260927 21:40:33.914342 10732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36451 } }
I20260927 21:40:33.914634 10732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:40:33.914827 10732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:40:33.915050 10732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:33.917725 10732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36451 } }
I20260927 21:40:33.918388 10732 leader_election.cc:302] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ebc5e3ac5544f73b426031e97ca3b4c; no voters:
I20260927 21:40:33.919070 10732 leader_election.cc:288] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:40:33.919229 10737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:33.919948 10737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [term 1 LEADER]: Becoming Leader. State: Replica: 0ebc5e3ac5544f73b426031e97ca3b4c, State: Running, Role: LEADER
I20260927 21:40:33.920639 10737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36451 } }
I20260927 21:40:33.921372 10732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:40:33.923470 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36451 } } }
I20260927 21:40:33.923852 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:33.924396 10739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ebc5e3ac5544f73b426031e97ca3b4c. Latest consensus state: current_term: 1 leader_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebc5e3ac5544f73b426031e97ca3b4c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36451 } } }
I20260927 21:40:33.924844 10739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:33.926754 10742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:40:33.932504 10742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:40:33.942168 10742 catalog_manager.cc:1347] Generated new cluster ID: 688c485d42d34ff4ac7f0e2d9f8c202c
I20260927 21:40:33.942304 10742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:40:33.954861 10742 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:40:33.955361 10742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:40:33.968804 10742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ebc5e3ac5544f73b426031e97ca3b4c: Generated new TSK 0
I20260927 21:40:33.969982 10742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:40:33.983757 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:36451
--builtin_ntp_servers=127.0.202.148:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:40:34.328609 10756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:34.328933 10756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:34.329059 10756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:34.336282 10756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:34.336442 10756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:34.336585 10756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:40:34.345768 10756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:34.347002 10756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:34.348812 10756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:34.361917 10761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:34.361953 10764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:34.361994 10762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:34.363399 10756 server_base.cc:1034] running on GCE node
I20260927 21:40:34.363961 10756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:34.364910 10756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:34.366106 10756 hybrid_clock.cc:648] HybridClock initialized: now 1790545234366063 us; error 31 us; skew 500 ppm
I20260927 21:40:34.366422 10756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:34.368788 10756 webserver.cc:460] Webserver started at http://127.0.202.129:46035/ using document root <none> and password file <none>
I20260927 21:40:34.369508 10756 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:34.369637 10756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:34.369944 10756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:34.372792 10756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "d2b615d71a004c469234a9fc56db4605"
format_stamp: "Formatted at 2026-09-27 21:40:34 on dist-test-slave-25dq"
server_key: "0490835c8de7b4ca6ba3b958d757821c"
server_key_iv: "9a794fd51e281164d852b1299ff57e8f"
server_key_version: "encryptionkey@0"
I20260927 21:40:34.373561 10756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "d2b615d71a004c469234a9fc56db4605"
format_stamp: "Formatted at 2026-09-27 21:40:34 on dist-test-slave-25dq"
server_key: "0490835c8de7b4ca6ba3b958d757821c"
server_key_iv: "9a794fd51e281164d852b1299ff57e8f"
server_key_version: "encryptionkey@0"
I20260927 21:40:34.378600 10756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:40:34.382086 10770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:34.383397 10756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:40:34.383574 10756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "d2b615d71a004c469234a9fc56db4605"
format_stamp: "Formatted at 2026-09-27 21:40:34 on dist-test-slave-25dq"
server_key: "0490835c8de7b4ca6ba3b958d757821c"
server_key_iv: "9a794fd51e281164d852b1299ff57e8f"
server_key_version: "encryptionkey@0"
I20260927 21:40:34.383805 10756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:34.389521 10756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:34.390300 10756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:34.390569 10756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:34.391494 10756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:34.393029 10756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:34.393155 10756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:34.393288 10756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:34.393373 10756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:34.427511 10756 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:43869
I20260927 21:40:34.427583 10882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:43869 every 8 connection(s)
I20260927 21:40:34.429071 10756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:40:34.430982 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 10756
I20260927 21:40:34.431219 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:40:34.432014 810 external_mini_cluster.cc:1428] Setting key 2ebaa976a7cd9ee041899372fd7da836
I20260927 21:40:34.435185 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:36451
--builtin_ntp_servers=127.0.202.148:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 21:40:34.438325 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:34.431793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:44751 (local address 127.0.202.190:36451)
0927 21:40:34.432263 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:34.432268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:34.433956 (+ 1688us) server_negotiation.cc:407] Connection header received
0927 21:40:34.435846 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:34.435854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:34.435932 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:34.436055 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:34.437489 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:34.437497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:34.437511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:34.437534 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:34.437585 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:34.437604 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:34.437733 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:34.437907 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 21:40:34.437974 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":248,"threads_started":1}
I20260927 21:40:34.439320 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:34.432568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:36451 (local address 127.0.202.129:44751)
0927 21:40:34.433785 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:34.433832 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:34.435572 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:34.436355 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:34.436365 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:34.436469 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:34.437181 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:40:34.437184 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:34.437225 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:40:34.437316 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:34.437704 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:34.437713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:34.437718 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:40:34.437939 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 21:40:34.438221 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":419,"threads_started":1}
I20260927 21:40:34.441035 10883 heartbeater.cc:347] Connected to a master server at 127.0.202.190:36451
I20260927 21:40:34.441347 10883 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:34.441946 10883 heartbeater.cc:511] Master 127.0.202.190:36451 requested a full tablet report, sending...
I20260927 21:40:34.443923 10697 ts_manager.cc:194] Registered new tserver with Master: d2b615d71a004c469234a9fc56db4605 (127.0.202.129:43869)
W20260927 21:40:34.779769 10887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:34.780191 10887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:34.780412 10887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:34.789330 10887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:34.789497 10887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:34.789896 10887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:40:34.800321 10887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:34.801924 10887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:34.803869 10887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:34.818142 10892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:34.818176 10895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:34.818221 10893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:34.819389 10887 server_base.cc:1034] running on GCE node
I20260927 21:40:34.820168 10887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:34.821417 10887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:34.822688 10887 hybrid_clock.cc:648] HybridClock initialized: now 1790545234822649 us; error 66 us; skew 500 ppm
I20260927 21:40:34.823077 10887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:34.825721 10887 webserver.cc:460] Webserver started at http://127.0.202.130:34495/ using document root <none> and password file <none>
I20260927 21:40:34.826447 10887 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:34.826592 10887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:34.826936 10887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:34.830132 10887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "2becc263a2604f60939549ed2ec8689d"
format_stamp: "Formatted at 2026-09-27 21:40:34 on dist-test-slave-25dq"
server_key: "acd1b0f6f88cf7004a3aa8d07b8fcc78"
server_key_iv: "d3c45acda875c6dba73dcd9db4e51638"
server_key_version: "encryptionkey@0"
I20260927 21:40:34.830950 10887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "2becc263a2604f60939549ed2ec8689d"
format_stamp: "Formatted at 2026-09-27 21:40:34 on dist-test-slave-25dq"
server_key: "acd1b0f6f88cf7004a3aa8d07b8fcc78"
server_key_iv: "d3c45acda875c6dba73dcd9db4e51638"
server_key_version: "encryptionkey@0"
I20260927 21:40:34.836261 10887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:40:34.839939 10901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:34.841675 10887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:34.841898 10887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "2becc263a2604f60939549ed2ec8689d"
format_stamp: "Formatted at 2026-09-27 21:40:34 on dist-test-slave-25dq"
server_key: "acd1b0f6f88cf7004a3aa8d07b8fcc78"
server_key_iv: "d3c45acda875c6dba73dcd9db4e51638"
server_key_version: "encryptionkey@0"
I20260927 21:40:34.842171 10887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:34.848506 10887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:34.849383 10887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:34.849706 10887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:34.850715 10887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:34.852403 10887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:34.852571 10887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:40:34.852717 10887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:34.852807 10887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:34.894701 10887 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:44449
I20260927 21:40:34.894733 11013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:44449 every 8 connection(s)
I20260927 21:40:34.896276 10887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:40:34.900377 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 10887
I20260927 21:40:34.900669 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:40:34.901194 810 external_mini_cluster.cc:1428] Setting key 86fb9adcd2a6dd2a601082fa51a5e652
I20260927 21:40:34.903995 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:36451
--builtin_ntp_servers=127.0.202.148:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 21:40:34.906035 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:34.899397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:49235 (local address 127.0.202.190:36451)
0927 21:40:34.899627 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:34.899633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:34.901555 (+ 1922us) server_negotiation.cc:407] Connection header received
0927 21:40:34.903559 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:34.903565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:34.903626 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:34.903752 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:34.905284 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:34.905289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:34.905294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:34.905313 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:34.905354 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:34.905366 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:34.905500 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:34.905695 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 21:40:34.905766 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:40:34.907105 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:34.900095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:36451 (local address 127.0.202.130:49235)
0927 21:40:34.901386 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:34.901427 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:34.903300 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:34.903997 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:34.904005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:34.904152 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:34.904957 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:40:34.904960 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:34.905006 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:40:34.905097 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:34.905471 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:34.905481 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:34.905489 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:40:34.905796 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 21:40:34.906110 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":423,"threads_started":1}
I20260927 21:40:34.908901 11014 heartbeater.cc:347] Connected to a master server at 127.0.202.190:36451
I20260927 21:40:34.909283 11014 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:34.909921 11014 heartbeater.cc:511] Master 127.0.202.190:36451 requested a full tablet report, sending...
I20260927 21:40:34.911649 10697 ts_manager.cc:194] Registered new tserver with Master: 2becc263a2604f60939549ed2ec8689d (127.0.202.130:44449)
W20260927 21:40:35.246968 11017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:35.247323 11017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:35.247493 11017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:35.255065 11017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:35.255214 11017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:35.255400 11017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:40:35.265136 11017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:35.266628 11017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:35.268522 11017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:35.282589 11022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:35.282665 11023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:35.282563 11025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:35.283075 11017 server_base.cc:1034] running on GCE node
I20260927 21:40:35.283972 11017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:35.285181 11017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:35.286458 11017 hybrid_clock.cc:648] HybridClock initialized: now 1790545235286384 us; error 61 us; skew 500 ppm
I20260927 21:40:35.286871 11017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:40:35.289553 11017 webserver.cc:460] Webserver started at http://127.0.202.131:40263/ using document root <none> and password file <none>
I20260927 21:40:35.290323 11017 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:35.290477 11017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:35.290822 11017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:35.293977 11017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "4b93ddb112d248d7aa87f26eba72ffc3"
format_stamp: "Formatted at 2026-09-27 21:40:35 on dist-test-slave-25dq"
server_key: "a01237eebd2a76727cd16fd5f64f607e"
server_key_iv: "5150a78cf466e7bcfc6ef1d8c5276c03"
server_key_version: "encryptionkey@0"
I20260927 21:40:35.294885 11017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "4b93ddb112d248d7aa87f26eba72ffc3"
format_stamp: "Formatted at 2026-09-27 21:40:35 on dist-test-slave-25dq"
server_key: "a01237eebd2a76727cd16fd5f64f607e"
server_key_iv: "5150a78cf466e7bcfc6ef1d8c5276c03"
server_key_version: "encryptionkey@0"
I20260927 21:40:35.300736 11017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 21:40:35.304720 11031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:35.306187 11017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:40:35.306387 11017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "4b93ddb112d248d7aa87f26eba72ffc3"
format_stamp: "Formatted at 2026-09-27 21:40:35 on dist-test-slave-25dq"
server_key: "a01237eebd2a76727cd16fd5f64f607e"
server_key_iv: "5150a78cf466e7bcfc6ef1d8c5276c03"
server_key_version: "encryptionkey@0"
I20260927 21:40:35.306646 11017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:35.312899 11017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:35.313738 11017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:35.314031 11017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:35.314968 11017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:35.316483 11017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:35.316601 11017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:35.316742 11017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:35.316836 11017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:35.351092 11017 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37097
I20260927 21:40:35.351187 11143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37097 every 8 connection(s)
I20260927 21:40:35.352473 11017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:40:35.359673 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 11017
I20260927 21:40:35.359872 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:40:35.360409 810 external_mini_cluster.cc:1428] Setting key 8a381dc497005c5856fb45ffdc654a54
I20260927 21:40:35.361842 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:35.355461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:32853 (local address 127.0.202.190:36451)
0927 21:40:35.355683 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:35.355690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:35.357347 (+ 1657us) server_negotiation.cc:407] Connection header received
0927 21:40:35.359211 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:35.359215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:35.359306 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:35.359429 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:35.361009 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:35.361015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:35.361020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:35.361041 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:35.361079 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:35.361095 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:35.361239 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:35.361467 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 21:40:35.361541 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:40:35.363315 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:35.356080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:36451 (local address 127.0.202.131:32853)
0927 21:40:35.357182 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:35.357228 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:35.358940 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:35.359674 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:35.359683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:35.359778 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:35.360676 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:40:35.360679 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:35.360725 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:40:35.360827 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:35.361212 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:35.361222 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:35.361230 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:40:35.361553 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 21:40:35.361817 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":345,"threads_started":1}
I20260927 21:40:35.364821 11144 heartbeater.cc:347] Connected to a master server at 127.0.202.190:36451
I20260927 21:40:35.365177 11144 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:35.365828 11144 heartbeater.cc:511] Master 127.0.202.190:36451 requested a full tablet report, sending...
I20260927 21:40:35.367295 10697 ts_manager.cc:194] Registered new tserver with Master: 4b93ddb112d248d7aa87f26eba72ffc3 (127.0.202.131:37097)
I20260927 21:40:35.376497 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:40:35.383039 10885 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:40:35.381345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.0.202.190:36451)
0927 21:40:35.381531 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:35.381536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:35.382190 (+ 654us) server_negotiation.cc:407] Connection header received
0927 21:40:35.382339 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:35.382342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:35.382394 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:35.382481 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:35.382780 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53934: BlockingRecv error: recv got EOF from 127.0.0.1:53934 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:40:35.384202 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 10756
I20260927 21:40:35.388044 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:35.385222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.0.202.129:43869)
0927 21:40:35.385721 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:35.385728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:35.385754 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:40:35.385810 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:35.385816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:35.386056 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:35.386264 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:35.387377 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:35.387386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:35.387403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:35.387426 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:35.387466 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:35.387505 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:35.387640 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:35.387732 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:40:35.387812 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260927 21:40:35.389294 10878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:35.472223 10883 heartbeater.cc:503] Master 127.0.202.190:36451 was elected leader, sending a full tablet report...
I20260927 21:40:35.473933 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 10756
I20260927 21:40:35.488641 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 10887
I20260927 21:40:35.492884 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:35.489670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.0.202.130:44449)
0927 21:40:35.490265 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:35.490273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:35.490307 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:40:35.490376 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:35.490383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:35.490648 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:35.490799 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:35.491976 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:35.491984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:35.491998 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:35.492013 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:35.492069 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:35.492086 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:35.492207 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:35.492545 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 21:40:35.492634 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":206,"threads_started":1}
I20260927 21:40:35.494251 11009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:35.578948 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 10887
I20260927 21:40:35.595319 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 11017
I20260927 21:40:35.599627 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:35.596272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.202.131:37097)
0927 21:40:35.597128 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:35.597139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:35.597181 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:40:35.597261 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:35.597271 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:35.597552 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:35.597729 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:35.598875 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:35.598881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:35.598893 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:40:35.598908 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:35.598945 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:40:35.598958 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:35.599141 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:35.599217 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 21:40:35.599318 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":397,"threads_started":1}
I20260927 21:40:35.600991 11139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:35.684409 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 11017
I20260927 21:40:35.700876 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 10666
I20260927 21:40:35.702476 10727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:35.807427 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 10666
2026-09-27T21:40:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2424 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:40:35 dist-test-slave-25dq krb5kdc[11162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:40:35 dist-test-slave-25dq krb5kdc[11162](info): set up 2 sockets
Sep 27 21:40:35 dist-test-slave-25dq krb5kdc[11162](info): commencing operation
krb5kdc: starting...
W20260927 21:40:36.965341 810 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:40:37 dist-test-slave-25dq krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:40:37.216817 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:34645
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:34645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:37.558207 11175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:37.558504 11175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:37.558602 11175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:37.566694 11175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:40:37.566818 11175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:37.566865 11175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:37.566915 11175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:40:37.566958 11175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:40:37.577068 11175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:34645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:37.578383 11175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:37.579632 11175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:37.592185 11180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:37.592228 11181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:37.592283 11183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:37.597908 11175 server_base.cc:1034] running on GCE node
I20260927 21:40:37.598784 11175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:37.600421 11175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:37.601704 11175 hybrid_clock.cc:648] HybridClock initialized: now 1790545237601623 us; error 63 us; skew 500 ppm
I20260927 21:40:37.602056 11175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:37 dist-test-slave-25dq krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:37.610046 11175 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:40:37.611953 11175 webserver.cc:460] Webserver started at http://127.0.202.190:43485/ using document root <none> and password file <none>
I20260927 21:40:37.612826 11175 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:37.612989 11175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:37.613368 11175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:37.616988 11175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "f411cd8d986544c799cab6d6e83e93c3"
format_stamp: "Formatted at 2026-09-27 21:40:37 on dist-test-slave-25dq"
server_key: "ee71c21127e56b6ea0320e2dbf2b9726"
server_key_iv: "e25ae8097cebe9a07814a9a01e759ebe"
server_key_version: "encryptionkey@0"
I20260927 21:40:37.617901 11175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "f411cd8d986544c799cab6d6e83e93c3"
format_stamp: "Formatted at 2026-09-27 21:40:37 on dist-test-slave-25dq"
server_key: "ee71c21127e56b6ea0320e2dbf2b9726"
server_key_iv: "e25ae8097cebe9a07814a9a01e759ebe"
server_key_version: "encryptionkey@0"
I20260927 21:40:37.624908 11175 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 21:40:37.629467 11190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:37.631126 11175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260927 21:40:37.631330 11175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "f411cd8d986544c799cab6d6e83e93c3"
format_stamp: "Formatted at 2026-09-27 21:40:37 on dist-test-slave-25dq"
server_key: "ee71c21127e56b6ea0320e2dbf2b9726"
server_key_iv: "e25ae8097cebe9a07814a9a01e759ebe"
server_key_version: "encryptionkey@0"
I20260927 21:40:37.631539 11175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:37.654052 11175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:37.658360 11175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:37.658653 11175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:37.678249 11175 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:34645
I20260927 21:40:37.678342 11241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:34645 every 8 connection(s)
I20260927 21:40:37.679699 11175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:40:37.681664 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 11175
I20260927 21:40:37.681890 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:40:37.682721 810 external_mini_cluster.cc:1428] Setting key c45be83b0dcf41448a1824079501bd0c
I20260927 21:40:37.684561 11242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:40:37 dist-test-slave-25dq krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:37.699277 11242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3: Bootstrap starting.
I20260927 21:40:37.703372 11242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:37.704528 11242 log.cc:824] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:37.707548 11242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3: No bootstrap required, opened a new log
I20260927 21:40:37.707738 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:37.684946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.0.202.190:34645)
0927 21:40:37.685757 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:37.685771 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:37.685814 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:40:37.687519 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:37.687541 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:37.687975 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:37.688416 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:37.690208 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:37.691688 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:37.693520 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:37.694433 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:37.694637 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:37.699780 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:37.699789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:37.699810 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:37.699868 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:37.703463 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:37.704179 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:37.704188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:37.704202 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:37.704449 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:37.704887 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:37.704892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:37.704895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:37.705476 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:37.705824 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:37.706271 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 21:40:37.706699 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":374,"threads_started":1}
I20260927 21:40:37.715452 11242 raft_consensus.cc:348] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411cd8d986544c799cab6d6e83e93c3" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34645 } }
I20260927 21:40:37.715865 11242 raft_consensus.cc:374] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:37.716032 11242 raft_consensus.cc:729] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f411cd8d986544c799cab6d6e83e93c3, State: Initialized, Role: FOLLOWER
I20260927 21:40:37.716706 11242 consensus_queue.cc:260] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411cd8d986544c799cab6d6e83e93c3" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34645 } }
I20260927 21:40:37.717002 11242 raft_consensus.cc:388] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:40:37.717171 11242 raft_consensus.cc:482] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:40:37.717361 11242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:37.719460 11242 raft_consensus.cc:504] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411cd8d986544c799cab6d6e83e93c3" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34645 } }
I20260927 21:40:37.719954 11242 leader_election.cc:302] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f411cd8d986544c799cab6d6e83e93c3; no voters:
I20260927 21:40:37.720638 11242 leader_election.cc:288] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:40:37.720824 11247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:37.722385 11247 raft_consensus.cc:686] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [term 1 LEADER]: Becoming Leader. State: Replica: f411cd8d986544c799cab6d6e83e93c3, State: Running, Role: LEADER
I20260927 21:40:37.722885 11242 sys_catalog.cc:564] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:40:37.722996 11247 consensus_queue.cc:237] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411cd8d986544c799cab6d6e83e93c3" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34645 } }
I20260927 21:40:37.726219 11248 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f411cd8d986544c799cab6d6e83e93c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411cd8d986544c799cab6d6e83e93c3" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34645 } } }
I20260927 21:40:37.726212 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f411cd8d986544c799cab6d6e83e93c3. Latest consensus state: current_term: 1 leader_uuid: "f411cd8d986544c799cab6d6e83e93c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411cd8d986544c799cab6d6e83e93c3" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 34645 } } }
I20260927 21:40:37.726827 11248 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:37.726917 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:40:37.728761 11256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:40:37.734407 11256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:40:37.743559 11256 catalog_manager.cc:1347] Generated new cluster ID: e8af723c30e54638a06264da4f80464d
I20260927 21:40:37.743759 11256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:40:37.770735 11256 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:40:37.772209 11256 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:40:37.780017 11256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3: Generated new TSK 0
I20260927 21:40:37.781175 11256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:40:37.848627 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34645
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:38.191092 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:38.191452 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:38.191622 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:38.199297 11268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:38.199477 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:38.199663 11268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:40:38.211557 11268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:38.213020 11268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:38.214476 11268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:38.227649 11273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:38.227636 11276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:38.227689 11274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:38.228884 11268 server_base.cc:1034] running on GCE node
I20260927 21:40:38.229485 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:38.230460 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:38.231688 11268 hybrid_clock.cc:648] HybridClock initialized: now 1790545238231644 us; error 28 us; skew 500 ppm
I20260927 21:40:38.232052 11268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:38 dist-test-slave-25dq krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:38.235445 11268 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:40:38.237035 11268 webserver.cc:460] Webserver started at http://127.0.202.129:35991/ using document root <none> and password file <none>
I20260927 21:40:38.237751 11268 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:38.237847 11268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:38.238183 11268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:38.241204 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "e9b31e785e8d415a9dd4c5a34da81a09"
format_stamp: "Formatted at 2026-09-27 21:40:38 on dist-test-slave-25dq"
server_key: "d439b5ba62c42327318879179c9da06c"
server_key_iv: "f77520cd20d7cf397151f0d41bf7b3f7"
server_key_version: "encryptionkey@0"
I20260927 21:40:38.241958 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "e9b31e785e8d415a9dd4c5a34da81a09"
format_stamp: "Formatted at 2026-09-27 21:40:38 on dist-test-slave-25dq"
server_key: "d439b5ba62c42327318879179c9da06c"
server_key_iv: "f77520cd20d7cf397151f0d41bf7b3f7"
server_key_version: "encryptionkey@0"
I20260927 21:40:38.247363 11268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:40:38.251144 11283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:38.252650 11268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:40:38.252825 11268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "e9b31e785e8d415a9dd4c5a34da81a09"
format_stamp: "Formatted at 2026-09-27 21:40:38 on dist-test-slave-25dq"
server_key: "d439b5ba62c42327318879179c9da06c"
server_key_iv: "f77520cd20d7cf397151f0d41bf7b3f7"
server_key_version: "encryptionkey@0"
I20260927 21:40:38.253046 11268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:38.275360 11268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:38.279495 11268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:38.279806 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:38.280881 11268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:38.282222 11268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:38.282323 11268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:38.282464 11268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:38.282534 11268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:38.315938 11268 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:37801
I20260927 21:40:38.315977 11395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:37801 every 8 connection(s)
I20260927 21:40:38.317263 11268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:40:38.325610 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 11268
I20260927 21:40:38.325832 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:40:38.326640 810 external_mini_cluster.cc:1428] Setting key fe139f9048ee090d1ba2533db6b78a46
Sep 27 21:40:38 dist-test-slave-25dq krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:38.338150 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:38.319788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:50711 (local address 127.0.202.190:34645)
0927 21:40:38.320206 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:38.320211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:38.321267 (+ 1056us) server_negotiation.cc:407] Connection header received
0927 21:40:38.323157 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:38.323163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:38.323255 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:38.323369 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:38.324937 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.325644 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.327827 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.328234 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.328416 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:38.332118 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:38.332123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:38.332126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:38.332157 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:38.334842 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:38.335745 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:38.335753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:38.335760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:38.335873 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:38.336328 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:38.336333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:38.336336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:38.336717 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:38.336866 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:38.337565 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 21:40:38.337776 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":194,"threads_started":1}
I20260927 21:40:38.339149 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:38.320377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:34645 (local address 127.0.202.129:50711)
0927 21:40:38.321111 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:38.321153 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:38.322883 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:38.323617 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:38.323630 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:38.324170 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:38.324778 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:38.324794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.325866 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.325874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:38.327669 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:38.327680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.328549 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.328557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:38.328806 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:38.329537 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:38.329578 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:38.331886 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:38.334992 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:38.335001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:38.335025 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:38.335620 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:38.335988 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:38.335993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:38.335997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:38.336195 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:38.336849 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:38.336857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:38.337196 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 21:40:38.337475 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 21:40:38.338007 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":265,"threads_started":1}
I20260927 21:40:38.340785 11396 heartbeater.cc:347] Connected to a master server at 127.0.202.190:34645
I20260927 21:40:38.341218 11396 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:38.341950 11396 heartbeater.cc:511] Master 127.0.202.190:34645 requested a full tablet report, sending...
I20260927 21:40:38.344331 11207 ts_manager.cc:194] Registered new tserver with Master: e9b31e785e8d415a9dd4c5a34da81a09 (127.0.202.129:37801)
I20260927 21:40:38.346577 11207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:50711
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:40:38.384608 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34645
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:38.721010 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:38.721307 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:38.721401 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:38.728864 11402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:38.729031 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:38.729151 11402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:40:38.740414 11402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:38.741806 11402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:38.743206 11402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:38.757619 11410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:38.757647 11408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:38.758002 11407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:38.758903 11402 server_base.cc:1034] running on GCE node
I20260927 21:40:38.759457 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:38.760426 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:38.761646 11402 hybrid_clock.cc:648] HybridClock initialized: now 1790545238761602 us; error 41 us; skew 500 ppm
I20260927 21:40:38.761962 11402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:38 dist-test-slave-25dq krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:38.765149 11402 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:40:38.766494 11402 webserver.cc:460] Webserver started at http://127.0.202.130:46035/ using document root <none> and password file <none>
I20260927 21:40:38.767164 11402 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:38.767256 11402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:38.767508 11402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:38.770366 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b"
format_stamp: "Formatted at 2026-09-27 21:40:38 on dist-test-slave-25dq"
server_key: "3e0ab5c3b1e229624206f3192b82486a"
server_key_iv: "34c342005bf5237df69207bc487b60ad"
server_key_version: "encryptionkey@0"
I20260927 21:40:38.771065 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b"
format_stamp: "Formatted at 2026-09-27 21:40:38 on dist-test-slave-25dq"
server_key: "3e0ab5c3b1e229624206f3192b82486a"
server_key_iv: "34c342005bf5237df69207bc487b60ad"
server_key_version: "encryptionkey@0"
I20260927 21:40:38.776163 11402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 21:40:38.779850 11417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:38.781509 11402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:38.781738 11402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b"
format_stamp: "Formatted at 2026-09-27 21:40:38 on dist-test-slave-25dq"
server_key: "3e0ab5c3b1e229624206f3192b82486a"
server_key_iv: "34c342005bf5237df69207bc487b60ad"
server_key_version: "encryptionkey@0"
I20260927 21:40:38.781970 11402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:38.809141 11402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:38.813640 11402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:38.813944 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:38.814898 11402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:38.816237 11402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:38.816363 11402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 21:40:38.816512 11402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:38.816591 11402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:38.850433 11402 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:37031
I20260927 21:40:38.850481 11529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:37031 every 8 connection(s)
I20260927 21:40:38.851763 11402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:40:38.858919 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 11402
I20260927 21:40:38.859143 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:40:38.859552 810 external_mini_cluster.cc:1428] Setting key 14209fe99bc80348682cd93301a86240
Sep 27 21:40:38 dist-test-slave-25dq krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:38.874137 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:38.854516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:40471 (local address 127.0.202.190:34645)
0927 21:40:38.854993 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:38.854998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:38.855808 (+ 810us) server_negotiation.cc:407] Connection header received
0927 21:40:38.857949 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:38.857953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:38.858029 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:38.858169 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:38.859829 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.860761 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.862779 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.863634 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.863780 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:38.867659 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:38.867669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:38.867676 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:38.867723 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:38.870693 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:38.871694 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:38.871698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:38.871700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:38.871818 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:38.872373 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:38.872376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:38.872378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:38.872628 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:38.872735 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:38.873702 (+ 967us) server_negotiation.cc:299] Negotiation successful
0927 21:40:38.873861 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":277,"threads_started":1}
I20260927 21:40:38.875792 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:38.854953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:34645 (local address 127.0.202.130:40471)
0927 21:40:38.855650 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:38.855688 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:38.857639 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:38.858380 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:38.858389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:38.858928 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:38.859616 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:38.859654 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.860912 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.860917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:38.862618 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:38.862635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:38.863885 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:38.863890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:38.864121 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:38.864895 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:38.864918 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:38.867396 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:38.870859 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:38.870872 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:38.870895 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:38.871563 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:38.871928 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:38.871935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:38.871942 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:38.872210 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:38.872764 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:38.872775 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:38.873270 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 21:40:38.873625 (+ 355us) client_negotiation.cc:240] Negotiation successful
0927 21:40:38.874228 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":280,"threads_started":1}
I20260927 21:40:38.877452 11530 heartbeater.cc:347] Connected to a master server at 127.0.202.190:34645
I20260927 21:40:38.877820 11530 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:38.878603 11530 heartbeater.cc:511] Master 127.0.202.190:34645 requested a full tablet report, sending...
I20260927 21:40:38.880254 11207 ts_manager.cc:194] Registered new tserver with Master: c4f02a5c3ad54c46ba3a9ec3d8edcd8b (127.0.202.130:37031)
I20260927 21:40:38.881167 11207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:40471
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:40:38.917299 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34645
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:40:39.265231 11536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:40:39.265588 11536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:40:39.265763 11536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:40:39.273077 11536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:40:39.273241 11536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:40:39.273433 11536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:40:39.283283 11536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:40:39.284614 11536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:40:39.286023 11536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:40:39.299674 11541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:39.299608 11542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:40:39.299767 11544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:40:39.300771 11536 server_base.cc:1034] running on GCE node
I20260927 21:40:39.301430 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:40:39.302382 11536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:40:39.303606 11536 hybrid_clock.cc:648] HybridClock initialized: now 1790545239303553 us; error 48 us; skew 500 ppm
I20260927 21:40:39.303913 11536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:40:39 dist-test-slave-25dq krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:40:39.307284 11536 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:40:39.308938 11536 webserver.cc:460] Webserver started at http://127.0.202.131:46241/ using document root <none> and password file <none>
I20260927 21:40:39.309634 11536 fs_manager.cc:359] Metadata directory not provided
I20260927 21:40:39.309774 11536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:40:39.310086 11536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:40:39.313001 11536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "4b1f367cccac40d2b51b1b6490e59f20"
format_stamp: "Formatted at 2026-09-27 21:40:39 on dist-test-slave-25dq"
server_key: "d7f0e18a576c348ae92da64ff52eb1d1"
server_key_iv: "d937f06ccae7d87ea98dedb2842712fc"
server_key_version: "encryptionkey@0"
I20260927 21:40:39.313777 11536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "4b1f367cccac40d2b51b1b6490e59f20"
format_stamp: "Formatted at 2026-09-27 21:40:39 on dist-test-slave-25dq"
server_key: "d7f0e18a576c348ae92da64ff52eb1d1"
server_key_iv: "d937f06ccae7d87ea98dedb2842712fc"
server_key_version: "encryptionkey@0"
I20260927 21:40:39.319519 11536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:40:39.323441 11551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:39.325009 11536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:40:39.325230 11536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "4b1f367cccac40d2b51b1b6490e59f20"
format_stamp: "Formatted at 2026-09-27 21:40:39 on dist-test-slave-25dq"
server_key: "d7f0e18a576c348ae92da64ff52eb1d1"
server_key_iv: "d937f06ccae7d87ea98dedb2842712fc"
server_key_version: "encryptionkey@0"
I20260927 21:40:39.325430 11536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:40:39.337872 11536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:40:39.342244 11536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:40:39.342537 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:40:39.343472 11536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:40:39.344944 11536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:40:39.345045 11536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:39.345201 11536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:40:39.345273 11536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:40:39.349864 11396 heartbeater.cc:503] Master 127.0.202.190:34645 was elected leader, sending a full tablet report...
I20260927 21:40:39.381125 11536 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:38309
I20260927 21:40:39.381225 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:38309 every 8 connection(s)
I20260927 21:40:39.382489 11536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:40:39.391829 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 11536
I20260927 21:40:39.392050 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:40:39.392588 810 external_mini_cluster.cc:1428] Setting key fddacba07d461ea0c3078c65df049bfb
Sep 27 21:40:39 dist-test-slave-25dq krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:40:39.404935 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.385188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:50719 (local address 127.0.202.190:34645)
0927 21:40:39.385703 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.385709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.386808 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 21:40:39.389050 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.389056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.389181 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.389319 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:39.390932 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.392018 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.394131 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.394759 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.394923 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.398834 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:39.398839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:39.398843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:39.398886 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:39.401741 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.402610 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.402614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.402617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.402740 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.403220 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.403224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.403226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.403496 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:39.403661 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.404495 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.404666 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":250,"threads_started":1}
I20260927 21:40:39.406230 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.385789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:34645 (local address 127.0.202.131:50719)
0927 21:40:39.386600 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:39.386647 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:39.388749 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:39.389563 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:39.389572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:39.390129 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:40:39.390743 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.390763 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.392218 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.392222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.393902 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.393920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.395041 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.395050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.395310 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.396096 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:40:39.396133 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:40:39.398613 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:40:39.401903 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:39.401912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:39.401926 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:39.402452 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:39.402845 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:40:39.402849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:40:39.402851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:40:39.403084 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:40:39.403651 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:40:39.403659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:40:39.404053 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 21:40:39.404409 (+ 356us) client_negotiation.cc:240] Negotiation successful
0927 21:40:39.404860 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":307,"threads_started":1}
I20260927 21:40:39.408133 11664 heartbeater.cc:347] Connected to a master server at 127.0.202.190:34645
I20260927 21:40:39.408552 11664 heartbeater.cc:464] Registering TS with master...
I20260927 21:40:39.409339 11664 heartbeater.cc:511] Master 127.0.202.190:34645 requested a full tablet report, sending...
I20260927 21:40:39.410844 11206 ts_manager.cc:194] Registered new tserver with Master: 4b1f367cccac40d2b51b1b6490e59f20 (127.0.202.131:38309)
I20260927 21:40:39.411791 11206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:50719
I20260927 21:40:39.421732 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:40:39.439906 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.426395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.0.202.190:34645)
0927 21:40:39.426592 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.426598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.427098 (+ 500us) server_negotiation.cc:407] Connection header received
0927 21:40:39.427275 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.427280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.427349 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.427433 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:39.429111 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.430105 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.431368 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.431834 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.431951 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.434239 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:39.434244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:39.434247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:39.434282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:39.437371 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.438015 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.438020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.438024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.438181 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.438542 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.438552 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.438555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.438903 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:39.439102 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.439443 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.439650 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:40:39.444164 11206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:40:39.451643 11206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:40:39.491480 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.481215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.0.202.129:37801)
0927 21:40:39.482296 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.482303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.482327 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:40:39.482409 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.482414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.482580 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.482811 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:39.483609 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.485263 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.488372 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.490075 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.490227 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.490456 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.490896 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.491109 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":774,"threads_started":1}
I20260927 21:40:39.491533 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.466260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.0.202.131:38309)
0927 21:40:39.478657 (+ 12397us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.478677 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.480602 (+ 1925us) server_negotiation.cc:407] Connection header received
0927 21:40:39.481961 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.481974 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.482252 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.482430 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:39.483659 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.485446 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.488832 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.490037 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.490120 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.490385 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.490994 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.491170 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12094,"thread_start_us":12422,"threads_started":1}
I20260927 21:40:39.491539 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.481266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:37801 (local address 127.0.0.1:50664)
0927 21:40:39.481928 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:39.481947 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:39.482055 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:39.482784 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:39.482793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:39.482823 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:39.483347 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.483358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.485407 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.485416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.488170 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.488183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.490300 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.490306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.490558 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.490654 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 21:40:39.490798 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:40:39.491007 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":323,"thread_start_us":339,"threads_started":1}
I20260927 21:40:39.492110 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.465466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37031 (local address 127.0.0.1:33352)
0927 21:40:39.479541 (+ 14075us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:39.479584 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:39.479807 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:39.482250 (+ 2443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:39.482258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:39.482290 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:39.482722 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.482738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.485551 (+ 2813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.485556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.489225 (+ 3669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.489240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.491169 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.491176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.491395 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.491431 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:40:39.491527 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:40:39.491698 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13809,"mutex_wait_us":58,"thread_start_us":14711,"threads_started":1}
I20260927 21:40:39.493031 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.465790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:38309 (local address 127.0.0.1:46426)
0927 21:40:39.480480 (+ 14690us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:39.480499 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:39.481733 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:39.482769 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:39.482776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:39.482811 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:39.483424 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.483438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.485591 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.485596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.488621 (+ 3025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.488648 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.490577 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.490583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.490772 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.490810 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:40:39.492478 (+ 1668us) client_negotiation.cc:240] Negotiation successful
0927 21:40:39.492622 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14559,"mutex_wait_us":145,"thread_start_us":13339,"threads_started":1}
I20260927 21:40:39.493954 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.465452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.0.202.130:37031)
0927 21:40:39.481074 (+ 15622us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.481082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.481107 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:40:39.481188 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.481194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.481410 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.482977 (+ 1567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:39.483265 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.485368 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.489407 (+ 4039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.490903 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.492518 (+ 1615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.492717 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.492791 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.492963 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15446,"thread_start_us":15304,"threads_started":1}
I20260927 21:40:39.496346 11331 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 (DEFAULT_TABLE table=test-table [id=5d56ab0368df47e787e64716a4407fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:39.497334 11465 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 (DEFAULT_TABLE table=test-table [id=5d56ab0368df47e787e64716a4407fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:39.497932 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:39.498510 11465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:39.499720 11599 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 (DEFAULT_TABLE table=test-table [id=5d56ab0368df47e787e64716a4407fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:40:39.501169 11599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:40:39.511613 11683 tablet_bootstrap.cc:492] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Bootstrap starting.
I20260927 21:40:39.515122 11683 tablet_bootstrap.cc:654] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:39.516103 11684 tablet_bootstrap.cc:492] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Bootstrap starting.
I20260927 21:40:39.518294 11683 log.cc:824] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:39.519881 11685 tablet_bootstrap.cc:492] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Bootstrap starting.
I20260927 21:40:39.520470 11684 tablet_bootstrap.cc:654] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:39.521647 11683 tablet_bootstrap.cc:492] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: No bootstrap required, opened a new log
I20260927 21:40:39.521931 11683 ts_tablet_manager.cc:1397] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 21:40:39.521927 11684 log.cc:824] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:39.523764 11685 tablet_bootstrap.cc:654] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Neither blocks nor log segments found. Creating new log.
I20260927 21:40:39.525048 11685 log.cc:824] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Log is configured to *not* fsync() on all Append() calls
I20260927 21:40:39.525267 11684 tablet_bootstrap.cc:492] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: No bootstrap required, opened a new log
I20260927 21:40:39.525550 11684 ts_tablet_manager.cc:1397] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 21:40:39.528095 11685 tablet_bootstrap.cc:492] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: No bootstrap required, opened a new log
I20260927 21:40:39.528379 11685 ts_tablet_manager.cc:1397] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 21:40:39.531299 11683 raft_consensus.cc:348] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.531690 11683 raft_consensus.cc:374] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:39.531814 11683 raft_consensus.cc:729] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f02a5c3ad54c46ba3a9ec3d8edcd8b, State: Initialized, Role: FOLLOWER
I20260927 21:40:39.532528 11683 consensus_queue.cc:260] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.533680 11530 heartbeater.cc:503] Master 127.0.202.190:34645 was elected leader, sending a full tablet report...
I20260927 21:40:39.534274 11683 ts_tablet_manager.cc:1428] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 21:40:39.537287 11684 raft_consensus.cc:348] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.537758 11684 raft_consensus.cc:374] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:39.537870 11684 raft_consensus.cc:729] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b31e785e8d415a9dd4c5a34da81a09, State: Initialized, Role: FOLLOWER
I20260927 21:40:39.537742 11685 raft_consensus.cc:348] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.538266 11685 raft_consensus.cc:374] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:40:39.538415 11685 raft_consensus.cc:729] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1f367cccac40d2b51b1b6490e59f20, State: Initialized, Role: FOLLOWER
I20260927 21:40:39.538552 11684 consensus_queue.cc:260] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.538986 11685 consensus_queue.cc:260] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.540585 11684 ts_tablet_manager.cc:1428] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 21:40:39.541644 11664 heartbeater.cc:503] Master 127.0.202.190:34645 was elected leader, sending a full tablet report...
I20260927 21:40:39.542326 11685 ts_tablet_manager.cc:1428] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260927 21:40:39.569638 11397 tablet.cc:2369] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:40:39.603888 11531 tablet.cc:2369] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:40:39.634145 11665 tablet.cc:2369] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:40:39.702740 11691 raft_consensus.cc:482] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:40:39.703115 11691 raft_consensus.cc:504] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.704924 11691 leader_election.cc:288] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f02a5c3ad54c46ba3a9ec3d8edcd8b (127.0.202.130:37031), e9b31e785e8d415a9dd4c5a34da81a09 (127.0.202.129:37801)
I20260927 21:40:39.711390 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.705334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37031 (local address 127.0.202.131:59293)
0927 21:40:39.705564 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:39.705585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:39.705710 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:39.706355 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:39.706362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:39.706394 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:39.706697 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.706714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.707921 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.707929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.709289 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.709305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.710420 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.710426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.710727 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.710757 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:40:39.710911 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 21:40:39.711063 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:40:39.711388 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.705542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:59293 (local address 127.0.202.130:37031)
0927 21:40:39.705841 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.705851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.705881 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:40:39.706046 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.706054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.706158 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.706280 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:39.706870 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.707772 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.709474 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.710262 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.710428 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.710507 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.710942 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.711098 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 21:40:39.712327 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.706014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:43853 (local address 127.0.202.129:37801)
0927 21:40:39.706261 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.706267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.706745 (+ 478us) server_negotiation.cc:407] Connection header received
0927 21:40:39.706967 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.706975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.707044 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.707154 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:40:39.707725 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.708575 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.710455 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.711327 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.711532 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.711609 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.711895 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.712040 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 21:40:39.712365 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.705743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:37801 (local address 127.0.202.131:43853)
0927 21:40:39.706632 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 21:40:39.706649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:40:39.706796 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:40:39.707175 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:40:39.707180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:40:39.707207 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:40:39.707562 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.707571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.708740 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.708746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.710297 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:40:39.710310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.711538 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.711544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:40:39.711731 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.711754 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:40:39.711858 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:40:39.711995 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":419,"threads_started":1}
I20260927 21:40:39.712983 11485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb78ad8e9ca4eefbe8db85cf01d5311" candidate_uuid: "4b1f367cccac40d2b51b1b6490e59f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" is_pre_election: true
I20260927 21:40:39.713521 11485 raft_consensus.cc:2454] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b1f367cccac40d2b51b1b6490e59f20 in term 0.
I20260927 21:40:39.713496 11351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb78ad8e9ca4eefbe8db85cf01d5311" candidate_uuid: "4b1f367cccac40d2b51b1b6490e59f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" is_pre_election: true
I20260927 21:40:39.713985 11351 raft_consensus.cc:2454] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b1f367cccac40d2b51b1b6490e59f20 in term 0.
I20260927 21:40:39.714282 11555 leader_election.cc:302] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1f367cccac40d2b51b1b6490e59f20, c4f02a5c3ad54c46ba3a9ec3d8edcd8b; no voters:
I20260927 21:40:39.714735 11691 raft_consensus.cc:2781] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:40:39.714903 11691 raft_consensus.cc:482] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:40:39.715077 11691 raft_consensus.cc:3037] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:39.717592 11691 raft_consensus.cc:504] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.718338 11691 leader_election.cc:288] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [CANDIDATE]: Term 1 election: Requested vote from peers c4f02a5c3ad54c46ba3a9ec3d8edcd8b (127.0.202.130:37031), e9b31e785e8d415a9dd4c5a34da81a09 (127.0.202.129:37801)
I20260927 21:40:39.718820 11485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb78ad8e9ca4eefbe8db85cf01d5311" candidate_uuid: "4b1f367cccac40d2b51b1b6490e59f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b"
I20260927 21:40:39.718829 11351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb78ad8e9ca4eefbe8db85cf01d5311" candidate_uuid: "4b1f367cccac40d2b51b1b6490e59f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b31e785e8d415a9dd4c5a34da81a09"
I20260927 21:40:39.719051 11485 raft_consensus.cc:3037] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:39.719061 11351 raft_consensus.cc:3037] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:40:39.721293 11351 raft_consensus.cc:2454] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b1f367cccac40d2b51b1b6490e59f20 in term 1.
I20260927 21:40:39.721524 11485 raft_consensus.cc:2454] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b1f367cccac40d2b51b1b6490e59f20 in term 1.
I20260927 21:40:39.721800 11555 leader_election.cc:302] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1f367cccac40d2b51b1b6490e59f20, e9b31e785e8d415a9dd4c5a34da81a09; no voters:
I20260927 21:40:39.722110 11691 raft_consensus.cc:2781] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:40:39.722644 11691 raft_consensus.cc:686] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 1 LEADER]: Becoming Leader. State: Replica: 4b1f367cccac40d2b51b1b6490e59f20, State: Running, Role: LEADER
I20260927 21:40:39.723265 11691 consensus_queue.cc:237] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } }
I20260927 21:40:39.728211 11207 catalog_manager.cc:5495] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b1f367cccac40d2b51b1b6490e59f20 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "4b1f367cccac40d2b51b1b6490e59f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b1f367cccac40d2b51b1b6490e59f20" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38309 } health_report { overall_health: HEALTHY } } }
I20260927 21:40:39.786404 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.780574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.0.202.131:38309)
0927 21:40:39.780810 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.780815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.780956 (+ 141us) server_negotiation.cc:407] Connection header received
0927 21:40:39.781142 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.781146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.781225 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.781375 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:40:39.781764 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.782536 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.784007 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.784472 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.784581 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.785200 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:40:39.785749 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:40:39.785868 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.786079 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.786173 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:40:39.795640 11485 raft_consensus.cc:1261] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 1 FOLLOWER]: Refusing update from remote peer 4b1f367cccac40d2b51b1b6490e59f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:39.795641 11351 raft_consensus.cc:1261] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 1 FOLLOWER]: Refusing update from remote peer 4b1f367cccac40d2b51b1b6490e59f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:40:39.796732 11693 consensus_queue.cc:1035] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b31e785e8d415a9dd4c5a34da81a09" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:39.797180 11691 consensus_queue.cc:1035] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f02a5c3ad54c46ba3a9ec3d8edcd8b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:40:39.811453 11698 mvcc.cc:205] Tried to move back new op lower bound from 7334073302192791552 to 7334073301911015424. Current Snapshot: MvccSnapshot[applied={T|T < 7334073302192791552}]
I20260927 21:40:39.822151 11207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:39.822492 11207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:40:39.827857 11207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f411cd8d986544c799cab6d6e83e93c3: Sending DeleteTablet for 3 replicas of tablet 8cb78ad8e9ca4eefbe8db85cf01d5311
I20260927 21:40:39.829253 11465 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:39 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:33352
I20260927 21:40:39.830333 11706 tablet_replica.cc:330] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: stopping tablet replica
I20260927 21:40:39.830806 11706 raft_consensus.cc:2229] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:39.830878 11331 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:39 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:50664
I20260927 21:40:39.831331 11706 raft_consensus.cc:2258] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:39.835208 11599 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:40:39 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:46426
I20260927 21:40:39.835819 11707 tablet_replica.cc:330] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: stopping tablet replica
I20260927 21:40:39.836313 11707 raft_consensus.cc:2229] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:40:39.836881 11707 raft_consensus.cc:2258] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:39.837682 11708 tablet_replica.cc:330] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: stopping tablet replica
I20260927 21:40:39.837898 11706 ts_tablet_manager.cc:1905] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:39.838086 11708 raft_consensus.cc:2229] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:40:39.838755 11708 raft_consensus.cc:2258] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:40:39.839190 11707 ts_tablet_manager.cc:1905] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:39.839583 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 11268
I20260927 21:40:39.847287 11706 ts_tablet_manager.cc:1918] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:39.847481 11706 log.cc:1196] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/8cb78ad8e9ca4eefbe8db85cf01d5311
I20260927 21:40:39.848097 11706 ts_tablet_manager.cc:1939] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P c4f02a5c3ad54c46ba3a9ec3d8edcd8b: Deleting consensus metadata
I20260927 21:40:39.848414 11707 ts_tablet_manager.cc:1918] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:39.848596 11707 log.cc:1196] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/8cb78ad8e9ca4eefbe8db85cf01d5311
I20260927 21:40:39.849167 11708 ts_tablet_manager.cc:1905] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:40:39.849213 11707 ts_tablet_manager.cc:1939] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P e9b31e785e8d415a9dd4c5a34da81a09: Deleting consensus metadata
I20260927 21:40:39.856694 11194 catalog_manager.cc:4841] TS c4f02a5c3ad54c46ba3a9ec3d8edcd8b (127.0.202.130:37031): tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 (table test-table [id=5d56ab0368df47e787e64716a4407fcc]) successfully deleted
I20260927 21:40:39.859522 11708 ts_tablet_manager.cc:1918] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:40:39.859740 11708 log.cc:1196] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/8cb78ad8e9ca4eefbe8db85cf01d5311
I20260927 21:40:39.860414 11708 ts_tablet_manager.cc:1939] T 8cb78ad8e9ca4eefbe8db85cf01d5311 P 4b1f367cccac40d2b51b1b6490e59f20: Deleting consensus metadata
I20260927 21:40:39.862174 11194 catalog_manager.cc:4841] TS e9b31e785e8d415a9dd4c5a34da81a09 (127.0.202.129:37801): tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 (table test-table [id=5d56ab0368df47e787e64716a4407fcc]) successfully deleted
I20260927 21:40:39.862287 11192 catalog_manager.cc:4841] TS 4b1f367cccac40d2b51b1b6490e59f20 (127.0.202.131:38309): tablet 8cb78ad8e9ca4eefbe8db85cf01d5311 (table test-table [id=5d56ab0368df47e787e64716a4407fcc]) successfully deleted
Sep 27 21:40:39 dist-test-slave-25dq krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:40:39.872089 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.841050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.0.202.129:37801)
0927 21:40:39.841358 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.841366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.852994 (+ 11628us) server_negotiation.cc:407] Connection header received
0927 21:40:39.853256 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.853261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.853349 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.853463 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:39.855271 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.856543 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.858632 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.859381 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.859495 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.865751 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:39.865765 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:39.865781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:39.865821 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:39.869038 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.869793 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.869802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.869805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.869949 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.870374 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.870381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.870385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.870766 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:39.871107 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.871423 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 21:40:39.871651 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260927 21:40:39.872910 11391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:39.965739 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 11268
I20260927 21:40:39.983672 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 11402
Sep 27 21:40:39 dist-test-slave-25dq krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:40:40.000623 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:39.984590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.0.202.130:37031)
0927 21:40:39.984783 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:39.984791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:39.984816 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:40:39.984878 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:39.984884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:39.984955 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:39.985063 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:39.986319 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.987188 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.988584 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:39.988985 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:39.989128 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:39.993383 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:39.993406 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:39.993429 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:39.993482 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:39.997575 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.998302 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.998313 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.998317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.998482 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:39.998867 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:39.998873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:39.998878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:39.999404 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:39.999732 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:39.999913 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 21:40:40.000170 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:40:40.001518 11525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:40.092445 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 11402
I20260927 21:40:40.110617 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 11536
Sep 27 21:40:40 dist-test-slave-25dq krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:40:40.130105 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:40:40.111534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.202.131:38309)
0927 21:40:40.111757 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:40:40.111765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:40:40.111788 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:40:40.111856 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:40:40.111861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:40:40.111988 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:40:40.112232 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:40:40.113727 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:40.114949 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:40.116462 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:40:40.117153 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:40:40.117291 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:40:40.121963 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:40:40.121980 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:40:40.122002 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:40:40.122048 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:40:40.126534 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:40.127308 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:40.127320 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:40.127323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:40.127524 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:40:40.127965 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:40:40.127970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:40:40.127974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:40:40.128556 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:40:40.128951 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0927 21:40:40.129383 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 21:40:40.129619 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 21:40:40.130997 11659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:40.252492 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 11536
I20260927 21:40:40.273672 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 11175
I20260927 21:40:40.274809 11237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:40:40.383960 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 11175
2026-09-27T21:40:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4578 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:40:40 dist-test-slave-25dq krb5kdc[11716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:40:40 dist-test-slave-25dq krb5kdc[11716](info): set up 2 sockets
Sep 27 21:40:40 dist-test-slave-25dq krb5kdc[11716](info): commencing operation
krb5kdc: starting...
W20260927 21:40:41.577337 810 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:40:41 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:40:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/rangeradmin_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/rangeradmin_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/rangerlookup_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/rangerlookup_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
I20260927 21:40:41.932946 810 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3Z3URy/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/postgres -l logfile start
2026-09-27 21:40:44.928 UTC [11746] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:40:44.928 UTC [11746] LOG: listening on IPv4 address "127.0.202.148", port 47085
2026-09-27 21:40:44.932 UTC [11746] LOG: listening on Unix socket "/tmp/.s.PGSQL.47085"
2026-09-27 21:40:44.955 UTC [11749] LOG: database system was shut down at 2026-09-27 21:40:43 UTC
2026-09-27 21:40:44.959 UTC [11746] LOG: database system is ready to accept connections
I20260927 21:40:45.949047 810 mini_postgres.cc:96] Postgres bound to 47085
2026-09-27 21:40:45.967 UTC [11757] FATAL: database "slave" does not exist
127.0.202.148:47085 - accepting connections
I20260927 21:40:45.967741 810 mini_ranger.cc:161] Starting Ranger...
I20260927 21:40:45.995098 810 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:40:46.670568 810 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:40:46.670733 810 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-admin
2026-09-27 21:40:46,994 [I] DB FLAVOR :POSTGRES
2026-09-27 21:40:46,994 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:40:46,995 [I] Checking connection..
2026-09-27 21:40:46,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:40:47,242 [I] Checking connection passed.
2026-09-27 21:40:47,243 [I] --------- Verifying version history table ---------
2026-09-27 21:40:47,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:40:47,504 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:40:47,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:40:47,743 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:40:47,743 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:40:47,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:40:47.992 UTC [11821] WARNING: there is no transaction in progress
2026-09-27 21:40:47,999 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:40:47,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:40:48,291 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:40:48,291 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:40:48,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:40:48,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:40:48,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-25dq.c.gcp-upstream.internal','N') ;"
2026-09-27 21:40:49,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:40:49,307 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:40:49,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:40:49,572 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:40:49,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:40:49,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:40:50,115 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:40:50,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:40:51.207 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.227 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.247 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.266 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.285 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.307 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.375 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.383 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.400 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.411 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.423 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.439 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.450 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.457 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.466 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:51.475 UTC [11929] WARNING: there is no transaction in progress
2026-09-27 21:40:52,258 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:40:52,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:40:52,518 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:40:52,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:40:52,783 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:40:52,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:40:53,055 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:40:53,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:40:53,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:40:53,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-25dq.c.gcp-upstream.internal';"
2026-09-27 21:40:53,817 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:40:53,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:40:54,090 [I] DB_PATCHES have already been applied
I20260927 21:40:54.095975 810 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-admin:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3Z3URy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:40:54.096071 810 mini_ranger.cc:193] Using host: 127.0.202.148
I20260927 21:40:54.111753 810 mini_ranger.cc:227] Ranger admin URL: http://127.0.202.148:44987
Sep 27, 2026 9:40:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:40:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:40:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:40:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:40:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:40:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:40:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:40:55.287321 810 mini_ranger.cc:160] Time spent starting Ranger: real 9.320s user 0.006s sys 0.038s
Sep 27, 2026 9:41:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:41:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 9:41:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:41:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:41:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:41:17.951565 810 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/rangerkms_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/rangerkms_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.202.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.202.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:41:18 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545278, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:41:18.123493 810 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:41:18.145641 810 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:41:19.262032 810 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:41:19.262218 810 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms
I20260927 21:41:19.262357 810 mini_ranger_kms.cc:227] Using postgres at 127.0.202.148:47085
2026-09-27 21:41:19,684 [I] DB FLAVOR :POSTGRES
2026-09-27 21:41:19,684 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:41:19,684 [I] Checking connection
2026-09-27 21:41:19,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:41:19,954 [I] connection success
2026-09-27 21:41:19,954 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:41:19,955 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:41:19,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:41:20,202 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:41:20,202 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:41:20,203 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:41:20,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:47085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:41:20,487 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:41:21.027529 810 mini_ranger_kms.cc:310] Created kms service
I20260927 21:41:21.230620 810 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:41:21.282456 810 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:41:21 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545278, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:41:21.709867 810 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:41:21.710173 810 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms:/tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:41:21.710213 810 mini_ranger_kms.cc:240] Using host: 127.0.202.148
I20260927 21:41:21.733441 810 mini_ranger_kms.cc:276] Ranger KMS PID: 12110
I20260927 21:41:21.733646 810 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.202.148:44655
Sep 27, 2026 9:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:41:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:41:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:41:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:41:24.180357 810 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.057s user 0.108s sys 0.334s
I20260927 21:41:24.180625 810 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:41:24.180838 810 mini_ranger_kms.cc:401] 127.0.202.148:44655/kms/v1/keys
Sep 27 21:41:24 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790545284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.202.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:41:24 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.202.148@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
Sep 27 21:41:25 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.202.148@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
Sep 27 21:41:27 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:41:27.629947 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:46067
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:46067
--ranger_config_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:44655/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:41:28.000483 12160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:41:28.000797 12160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:41:28.000954 12160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:41:28.008399 12160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:41:28.008517 12160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:41:28.008595 12160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:41:28.008684 12160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:41:28.008787 12160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:41:28.019613 12160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.202.148:44655/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:46067
--ranger_config_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:46067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:41:28.021024 12160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:41:28.022414 12160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:41:28.035430 12165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:41:28.035742 12166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:41:28.036257 12168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:41:28.036677 12160 server_base.cc:1034] running on GCE node
I20260927 21:41:28.037478 12160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:41:28.039003 12160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:41:28.040251 12160 hybrid_clock.cc:648] HybridClock initialized: now 1790545288040196 us; error 54 us; skew 500 ppm
I20260927 21:41:28.040670 12160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:41:28 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:41:28.048379 12160 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:41:28.050190 12160 webserver.cc:460] Webserver started at http://127.0.202.190:38791/ using document root <none> and password file <none>
I20260927 21:41:28.050889 12160 fs_manager.cc:359] Metadata directory not provided
I20260927 21:41:28.051021 12160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:41:28.051342 12160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:41:28 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:41:28.118685 12160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "6cab4dfe0a424c308dadf5b311c5f6a7"
format_stamp: "Formatted at 2026-09-27 21:41:28 on dist-test-slave-25dq"
server_key: "dafb9dce527b01bdd8e8f402507b88f2"
server_key_iv: "4a917d8de3e011afe8d7885868adf880"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:28.119625 12160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "6cab4dfe0a424c308dadf5b311c5f6a7"
format_stamp: "Formatted at 2026-09-27 21:41:28 on dist-test-slave-25dq"
server_key: "dafb9dce527b01bdd8e8f402507b88f2"
server_key_iv: "4a917d8de3e011afe8d7885868adf880"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:28.125765 12160 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:41:28.146893 12175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:28.148602 12160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:41:28.148800 12160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "6cab4dfe0a424c308dadf5b311c5f6a7"
format_stamp: "Formatted at 2026-09-27 21:41:28 on dist-test-slave-25dq"
server_key: "dafb9dce527b01bdd8e8f402507b88f2"
server_key_iv: "4a917d8de3e011afe8d7885868adf880"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:28.149032 12160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:41:28.175009 12160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:41:28.179955 12160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:41:28.180338 12160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:41:28.200910 12160 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:46067
I20260927 21:41:28.200935 12226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:46067 every 8 connection(s)
I20260927 21:41:28.202798 12160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:41:28.207083 12227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:41:28.210472 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 12160
I20260927 21:41:28.210685 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:41:28.211367 810 external_mini_cluster.cc:1428] Setting key f0d1b7e478512b97f2c2de287a51a2d8
I20260927 21:41:28.221253 12227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7: Bootstrap starting.
I20260927 21:41:28.226174 12227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7: Neither blocks nor log segments found. Creating new log.
I20260927 21:41:28.227551 12227 log.cc:824] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:41:28 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:41:28.231693 12227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7: No bootstrap required, opened a new log
I20260927 21:41:28.238931 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:28.213726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.0.202.190:46067)
0927 21:41:28.214569 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:28.214582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:28.214627 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:41:28.216414 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:28.216442 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:28.216892 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:28.217273 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:28.219087 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:28.220577 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:28.222899 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:28.223648 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:28.223783 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:28.229197 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:28.229210 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:28.229233 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:28.229285 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:28.233559 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:28.234327 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:28.234337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:28.234346 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:28.234554 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:28.235007 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:28.235013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:28.235017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:28.235817 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:28.236148 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:28.236487 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 21:41:28.236965 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":392,"threads_started":1}
I20260927 21:41:28.243250 12227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46067 } }
I20260927 21:41:28.243649 12227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:41:28.243799 12227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cab4dfe0a424c308dadf5b311c5f6a7, State: Initialized, Role: FOLLOWER
I20260927 21:41:28.244580 12227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46067 } }
I20260927 21:41:28.244832 12227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:41:28.244982 12227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:41:28.245167 12227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:41:28.247668 12227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46067 } }
I20260927 21:41:28.248266 12227 leader_election.cc:302] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cab4dfe0a424c308dadf5b311c5f6a7; no voters:
I20260927 21:41:28.249085 12227 leader_election.cc:288] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:41:28.249253 12232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:41:28.250034 12232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6cab4dfe0a424c308dadf5b311c5f6a7, State: Running, Role: LEADER
I20260927 21:41:28.250663 12232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46067 } }
I20260927 21:41:28.251437 12227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:41:28.254376 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cab4dfe0a424c308dadf5b311c5f6a7. Latest consensus state: current_term: 1 leader_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46067 } } }
I20260927 21:41:28.255898 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:41:28.255465 12233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab4dfe0a424c308dadf5b311c5f6a7" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 46067 } } }
I20260927 21:41:28.256505 12233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:41:28.258023 12237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:41:28.264094 12237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:41:28.265779 12227 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:41:28.284883 12237 catalog_manager.cc:1347] Generated new cluster ID: 9a5e7bc922d24511b10337b3576ed3d0
I20260927 21:41:28.285110 12237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:41:28.313144 12237 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:41:28.315207 12237 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:41:28.328450 12237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7: Generated new TSK 0
I20260927 21:41:28.329808 12237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:41:29.101729 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:44655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46067
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:41:29 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790545289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:41:29.518926 12272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:41:29.519471 12272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:41:29.519728 12272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:41:29.528733 12272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:41:29.528890 12272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:41:29.529065 12272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:41:29.540531 12272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.202.148:44655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:41:29.541990 12272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:41:29.543412 12272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:41:29.560622 12283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:41:29.561535 12282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:41:29.561914 12285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:41:29.563526 12272 server_base.cc:1034] running on GCE node
I20260927 21:41:29.564327 12272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:41:29.566000 12272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:41:29.567332 12272 hybrid_clock.cc:648] HybridClock initialized: now 1790545289567217 us; error 104 us; skew 500 ppm
I20260927 21:41:29.567771 12272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:41:29 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:41:29.572619 12272 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:41:29.574369 12272 webserver.cc:460] Webserver started at http://127.0.202.129:34669/ using document root <none> and password file <none>
I20260927 21:41:29.575124 12272 fs_manager.cc:359] Metadata directory not provided
I20260927 21:41:29.575275 12272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:41:29.575664 12272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:41:29 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:41:29.595209 12272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "0a617794e7ac4ed884baa7f785c2145c"
format_stamp: "Formatted at 2026-09-27 21:41:29 on dist-test-slave-25dq"
server_key: "1985b55b7e5a6643ad434923ee87afc3"
server_key_iv: "f6ab81bd6b6d37a7e3acbda503041c73"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:29.598006 12272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "0a617794e7ac4ed884baa7f785c2145c"
format_stamp: "Formatted at 2026-09-27 21:41:29 on dist-test-slave-25dq"
server_key: "1985b55b7e5a6643ad434923ee87afc3"
server_key_iv: "f6ab81bd6b6d37a7e3acbda503041c73"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:29.611832 12272 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.012s
I20260927 21:41:29.641614 12292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:29.644001 12272 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.008s
I20260927 21:41:29.644236 12272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "0a617794e7ac4ed884baa7f785c2145c"
format_stamp: "Formatted at 2026-09-27 21:41:29 on dist-test-slave-25dq"
server_key: "1985b55b7e5a6643ad434923ee87afc3"
server_key_iv: "f6ab81bd6b6d37a7e3acbda503041c73"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:29.644546 12272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:41:29.680800 12272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:41:29.685941 12272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:41:29.686359 12272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:41:29.688953 12272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:41:29.690589 12272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:41:29.690688 12272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:29.690812 12272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:41:29.690896 12272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:29.751580 12272 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:37045
I20260927 21:41:29.751638 12404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:37045 every 8 connection(s)
I20260927 21:41:29.754599 12272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:41:29.762398 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 12272
I20260927 21:41:29.762652 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:41:29.763435 810 external_mini_cluster.cc:1428] Setting key 33af9f7154704c6987696309c4ad85e9
Sep 27 21:41:29 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:41:29.800385 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:29.759200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:45637 (local address 127.0.202.190:46067)
0927 21:41:29.773595 (+ 14395us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:29.773604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:29.773630 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:41:29.773701 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:29.773709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:29.773814 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:29.773929 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:29.776880 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:29.778059 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:29.786142 (+ 8083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:29.786820 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:29.786944 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:29.791988 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:29.791996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:29.792001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:29.792046 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:29.795677 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:29.796798 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:29.796804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:29.796808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:29.796970 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:29.797570 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:29.797576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:29.797579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:29.797962 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:29.798090 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:29.799573 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0927 21:41:29.799839 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14244,"thread_start_us":15167,"threads_started":1}
I20260927 21:41:29.803068 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:29.758530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46067 (local address 127.0.202.129:45637)
0927 21:41:29.768991 (+ 10461us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:29.769069 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:29.772228 (+ 3159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:29.774988 (+ 2760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:29.775005 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:29.775725 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:41:29.776639 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:29.776668 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:29.778311 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:29.778329 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:29.780529 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:29.780548 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:29.787391 (+ 6843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:29.787401 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:29.787771 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:29.788717 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:41:29.788752 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:41:29.791644 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:41:29.795905 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:41:29.795916 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:41:29.795938 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:41:29.796636 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:41:29.797139 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:41:29.797144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:41:29.797147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:41:29.797434 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:41:29.798182 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:41:29.798192 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:41:29.798712 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 21:41:29.800976 (+ 2264us) client_negotiation.cc:240] Negotiation successful
0927 21:41:29.801590 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10071,"thread_start_us":9820,"threads_started":1}
I20260927 21:41:29.806617 12405 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46067
I20260927 21:41:29.807013 12405 heartbeater.cc:464] Registering TS with master...
I20260927 21:41:29.807943 12405 heartbeater.cc:511] Master 127.0.202.190:46067 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:41:29.810752 12192 ts_manager.cc:194] Registered new tserver with Master: 0a617794e7ac4ed884baa7f785c2145c (127.0.202.129:37045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:41:29.812983 12192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:45637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:41:29.859491 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:44655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46067
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:41:29 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
Sep 27 21:41:30 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
W20260927 21:41:30.344473 12411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:41:30.344974 12411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:41:30.345166 12411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:41:30.357076 12411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:41:30.357319 12411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:41:30.357565 12411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:41:30.373076 12411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.202.148:44655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:41:30.375025 12411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:41:30.376827 12411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:41:30.401027 12419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:41:30.404163 12418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:41:30.408591 12411 server_base.cc:1034] running on GCE node
W20260927 21:41:30.409495 12421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:41:30.410363 12411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:41:30.411717 12411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:41:30.413015 12411 hybrid_clock.cc:648] HybridClock initialized: now 1790545290412946 us; error 58 us; skew 500 ppm
I20260927 21:41:30.413509 12411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:41:30 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:41:30.417959 12411 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:41:30.426365 12411 webserver.cc:460] Webserver started at http://127.0.202.130:34041/ using document root <none> and password file <none>
I20260927 21:41:30.427294 12411 fs_manager.cc:359] Metadata directory not provided
I20260927 21:41:30.427461 12411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:41:30.427850 12411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:41:30 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:41:30.447389 12411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "489c0ae064364e70bfa30a2324891d96"
format_stamp: "Formatted at 2026-09-27 21:41:30 on dist-test-slave-25dq"
server_key: "d0bebf56d928e60e632b7949b5c12269"
server_key_iv: "f59aae5f5650039cfabf5c87b267eb90"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:30.449810 12411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "489c0ae064364e70bfa30a2324891d96"
format_stamp: "Formatted at 2026-09-27 21:41:30 on dist-test-slave-25dq"
server_key: "d0bebf56d928e60e632b7949b5c12269"
server_key_iv: "f59aae5f5650039cfabf5c87b267eb90"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:30.474773 12411 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.007s sys 0.015s
I20260927 21:41:30.513105 12431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:30.517416 12411 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.005s sys 0.016s
I20260927 21:41:30.517731 12411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "489c0ae064364e70bfa30a2324891d96"
format_stamp: "Formatted at 2026-09-27 21:41:30 on dist-test-slave-25dq"
server_key: "d0bebf56d928e60e632b7949b5c12269"
server_key_iv: "f59aae5f5650039cfabf5c87b267eb90"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:30.518110 12411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:41:30.571002 12411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:41:30.576200 12411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:41:30.576565 12411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:41:30.578109 12411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:41:30.580036 12411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:41:30.580147 12411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:30.580259 12411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:41:30.580348 12411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 21:41:30.626281 12411 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:42705
I20260927 21:41:30.626309 12547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:42705 every 8 connection(s)
I20260927 21:41:30.628006 12411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:41:30.636129 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 12411
I20260927 21:41:30.636359 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:41:30.637068 810 external_mini_cluster.cc:1428] Setting key fa94957cf302cc24490153639feb0843
Sep 27 21:41:30 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:41:30.659965 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:30.632667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:49709 (local address 127.0.202.190:46067)
0927 21:41:30.634947 (+ 2280us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:30.634955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:30.638584 (+ 3629us) server_negotiation.cc:407] Connection header received
0927 21:41:30.641284 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:30.641291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:30.641387 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:30.641524 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:30.643735 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:30.644986 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:30.648218 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:30.649041 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:30.649196 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:30.653609 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:30.653616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:30.653621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:30.653668 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:30.656583 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:30.657395 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:30.657401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:30.657405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:30.657540 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:30.658092 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:30.658098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:30.658101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:30.658394 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:30.658579 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:30.659337 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 21:41:30.659586 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2105,"thread_start_us":2007,"threads_started":1}
I20260927 21:41:30.661451 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:30.631966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46067 (local address 127.0.202.130:49709)
0927 21:41:30.638405 (+ 6439us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:30.638466 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:30.640881 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:30.641850 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:30.641865 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:30.642570 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:41:30.643494 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:30.643528 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:30.645184 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:30.645193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:30.647981 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:30.648030 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:30.649344 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:30.649354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:30.649720 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:30.650548 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:41:30.650580 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:41:30.653291 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:41:30.656737 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:41:30.656747 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:41:30.656767 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:41:30.657269 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:41:30.657657 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:41:30.657662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:41:30.657666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:41:30.657877 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:41:30.658555 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:41:30.658565 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:41:30.658936 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 21:41:30.659238 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 21:41:30.659788 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6119,"thread_start_us":5881,"threads_started":1}
I20260927 21:41:30.663177 12548 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46067
I20260927 21:41:30.663537 12548 heartbeater.cc:464] Registering TS with master...
I20260927 21:41:30.664644 12548 heartbeater.cc:511] Master 127.0.202.190:46067 requested a full tablet report, sending...
I20260927 21:41:30.666165 12192 ts_manager.cc:194] Registered new tserver with Master: 489c0ae064364e70bfa30a2324891d96 (127.0.202.130:42705)
I20260927 21:41:30.666970 12192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:49709
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:41:30.709408 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:44655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46067
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:41:30.817149 12405 heartbeater.cc:503] Master 127.0.202.190:46067 was elected leader, sending a full tablet report...
W20260927 21:41:31.084581 12554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:41:31.084870 12554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:41:31.085076 12554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:41:31.092437 12554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:41:31.092593 12554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:41:31.092790 12554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:41:31.103153 12554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.202.148:44655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:41:31.104625 12554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:41:31.105962 12554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:41:31.120024 12559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:41:31.120079 12560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:41:31.120039 12562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:41:31.121352 12554 server_base.cc:1034] running on GCE node
I20260927 21:41:31.121989 12554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:41:31.123085 12554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:41:31.124402 12554 hybrid_clock.cc:648] HybridClock initialized: now 1790545291124300 us; error 53 us; skew 500 ppm
I20260927 21:41:31.124763 12554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:41:31 dist-test-slave-25dq krb5kdc[11716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:41:31.128434 12554 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:41:31.130167 12554 webserver.cc:460] Webserver started at http://127.0.202.131:41779/ using document root <none> and password file <none>
I20260927 21:41:31.130892 12554 fs_manager.cc:359] Metadata directory not provided
I20260927 21:41:31.131038 12554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:41:31.131356 12554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:41:31 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:41:31.149742 12554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "5dc3582e23914c8d9d7f6300caecc64a"
format_stamp: "Formatted at 2026-09-27 21:41:31 on dist-test-slave-25dq"
server_key: "264c97a97c9bd3d7b1766130f64c89b2"
server_key_iv: "cb42795ebeff1bd78f7625a9fa2a600d"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:31.150612 12554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "5dc3582e23914c8d9d7f6300caecc64a"
format_stamp: "Formatted at 2026-09-27 21:41:31 on dist-test-slave-25dq"
server_key: "264c97a97c9bd3d7b1766130f64c89b2"
server_key_iv: "cb42795ebeff1bd78f7625a9fa2a600d"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:31.156988 12554 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:41:31.173978 12569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:31.175722 12554 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:41:31.175930 12554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "5dc3582e23914c8d9d7f6300caecc64a"
format_stamp: "Formatted at 2026-09-27 21:41:31 on dist-test-slave-25dq"
server_key: "264c97a97c9bd3d7b1766130f64c89b2"
server_key_iv: "cb42795ebeff1bd78f7625a9fa2a600d"
server_key_version: "kuduclusterkey@0"
I20260927 21:41:31.176178 12554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:41:31.198123 12554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:41:31.203140 12554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:41:31.203508 12554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:41:31.204680 12554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:41:31.206355 12554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:41:31.206480 12554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:31.206650 12554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:41:31.206730 12554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:41:31.248083 12554 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:44989
I20260927 21:41:31.248189 12681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:44989 every 8 connection(s)
I20260927 21:41:31.249571 12554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:41:31.258845 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 12554
I20260927 21:41:31.259063 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:41:31.259496 810 external_mini_cluster.cc:1428] Setting key 0c66bd8356b1f9fd9b5c4b1adc66a398
Sep 27 21:41:31 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:41:31.272568 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.252668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:57857 (local address 127.0.202.190:46067)
0927 21:41:31.253318 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.253324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.254300 (+ 976us) server_negotiation.cc:407] Connection header received
0927 21:41:31.256186 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.256193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.256331 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.256460 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:31.257916 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.258659 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.260633 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.261439 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.261583 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.265633 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:31.265639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:31.265643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:31.265680 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:31.269271 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.270147 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.270152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.270154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.270292 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.270766 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.270771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.270775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.271087 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:31.271206 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.271921 (+ 715us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.272165 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":370,"threads_started":1}
I20260927 21:41:31.273546 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.253203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:46067 (local address 127.0.202.131:57857)
0927 21:41:31.254023 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:31.254069 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:31.255886 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:31.256707 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:31.256716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:31.257165 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:41:31.257739 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.257756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.258827 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.258831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.260461 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.260482 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.261757 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.261766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.262110 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.263046 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:41:31.263076 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:41:31.265388 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:41:31.269460 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:41:31.269468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:41:31.269483 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:41:31.270014 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:41:31.270430 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:41:31.270433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:41:31.270436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:41:31.270657 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:41:31.271214 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:41:31.271220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:41:31.271590 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 21:41:31.271834 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 21:41:31.272250 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":299,"threads_started":1}
I20260927 21:41:31.275662 12682 heartbeater.cc:347] Connected to a master server at 127.0.202.190:46067
I20260927 21:41:31.276173 12682 heartbeater.cc:464] Registering TS with master...
I20260927 21:41:31.277045 12682 heartbeater.cc:511] Master 127.0.202.190:46067 requested a full tablet report, sending...
I20260927 21:41:31.278707 12191 ts_manager.cc:194] Registered new tserver with Master: 5dc3582e23914c8d9d7f6300caecc64a (127.0.202.131:44989)
I20260927 21:41:31.279717 12191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:57857
I20260927 21:41:31.289443 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:41:31.308187 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.294191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.0.202.190:46067)
0927 21:41:31.294435 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.294446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.294812 (+ 366us) server_negotiation.cc:407] Connection header received
0927 21:41:31.294975 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.294978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.295055 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.295139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:31.296805 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.297639 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.299091 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.299508 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.299604 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.302091 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:31.302096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:31.302100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:31.302132 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:31.305266 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.306331 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.306335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.306337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.306456 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.306782 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.306785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.306787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.307051 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:31.307236 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.307590 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.307755 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:41:31.313186 12191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:41:31.320876 12191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:41:31.359929 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.337137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:37045 (local address 127.0.0.1:36804)
0927 21:41:31.346239 (+ 9102us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:31.346318 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:31.346578 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:31.350119 (+ 3541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:31.350134 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:31.350168 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:41:31.351145 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.351158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.353878 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.353884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.356617 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.356632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.358991 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.358998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.359187 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.359235 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:41:31.359367 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:41:31.359571 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8721,"thread_start_us":12837,"threads_started":1}
I20260927 21:41:31.360718 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.337399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.0.202.129:37045)
0927 21:41:31.348807 (+ 11408us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.348822 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.348869 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:41:31.348988 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.348996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.349360 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.349612 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:41:31.351387 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.353040 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.356823 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.358287 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.358431 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.358699 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.360189 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.360388 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10751,"thread_start_us":10904,"threads_started":1}
I20260927 21:41:31.361423 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.337332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:44989 (local address 127.0.0.1:51480)
0927 21:41:31.350757 (+ 13425us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:31.350780 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:31.350913 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:31.352526 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:31.352532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:31.352637 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:41:31.353066 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.353078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.355536 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.355542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.358409 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.358424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.360546 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.360553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.360754 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.360794 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:41:31.360887 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:41:31.361035 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12803,"thread_start_us":9521,"threads_started":1}
I20260927 21:41:31.362203 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.338182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.0.202.131:44989)
0927 21:41:31.339217 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.339232 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.351078 (+ 11846us) server_negotiation.cc:407] Connection header received
0927 21:41:31.351910 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.351918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.352247 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.353301 (+ 1054us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:41:31.353662 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.355358 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.358584 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.360033 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.360159 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.360398 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.361825 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.361974 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":477,"threads_started":1}
I20260927 21:41:31.365368 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.350784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:42705 (local address 127.0.0.1:55944)
0927 21:41:31.356871 (+ 6087us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:31.356889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:31.357057 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:31.357962 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:31.357974 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:31.358003 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:41:31.358464 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.358476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.360447 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.360455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.362661 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.362676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.364624 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.364630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.364802 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.364834 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:41:31.364930 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:41:31.365062 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5923,"thread_start_us":5741,"threads_started":1}
I20260927 21:41:31.365526 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.350786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.0.202.130:42705)
0927 21:41:31.351808 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.351819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.357022 (+ 5203us) server_negotiation.cc:407] Connection header received
0927 21:41:31.357217 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.357228 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.357455 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.357683 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:41:31.359073 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.360289 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.362868 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.364413 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.364558 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.364758 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.365009 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.365174 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":752,"threads_started":1}
I20260927 21:41:31.365837 12340 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb0680324534e04ae82e1504a6222a0 (DEFAULT_TABLE table=test-table [id=2ba740ff982f40b69629a7c2f76d4789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:41:31.365839 12617 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb0680324534e04ae82e1504a6222a0 (DEFAULT_TABLE table=test-table [id=2ba740ff982f40b69629a7c2f76d4789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:41:31.367139 12340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb0680324534e04ae82e1504a6222a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:41:31.367225 12617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb0680324534e04ae82e1504a6222a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:41:31.369148 12483 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb0680324534e04ae82e1504a6222a0 (DEFAULT_TABLE table=test-table [id=2ba740ff982f40b69629a7c2f76d4789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:41:31.370545 12483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb0680324534e04ae82e1504a6222a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:41:31.387959 12703 tablet_bootstrap.cc:492] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Bootstrap starting.
I20260927 21:41:31.390697 12701 tablet_bootstrap.cc:492] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Bootstrap starting.
I20260927 21:41:31.391424 12703 tablet_bootstrap.cc:654] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Neither blocks nor log segments found. Creating new log.
I20260927 21:41:31.393298 12703 log.cc:824] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:41:31.395658 12701 tablet_bootstrap.cc:654] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Neither blocks nor log segments found. Creating new log.
I20260927 21:41:31.397010 12702 tablet_bootstrap.cc:492] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Bootstrap starting.
I20260927 21:41:31.399840 12702 tablet_bootstrap.cc:654] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Neither blocks nor log segments found. Creating new log.
I20260927 21:41:31.401180 12702 log.cc:824] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Log is configured to *not* fsync() on all Append() calls
I20260927 21:41:31.401182 12701 log.cc:824] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:41:31.404115 12703 tablet_bootstrap.cc:492] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: No bootstrap required, opened a new log
I20260927 21:41:31.404525 12703 ts_tablet_manager.cc:1397] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20260927 21:41:31.408206 12702 tablet_bootstrap.cc:492] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: No bootstrap required, opened a new log
I20260927 21:41:31.408558 12702 ts_tablet_manager.cc:1397] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 21:41:31.410502 12701 tablet_bootstrap.cc:492] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: No bootstrap required, opened a new log
I20260927 21:41:31.410777 12701 ts_tablet_manager.cc:1397] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20260927 21:41:31.419082 12703 raft_consensus.cc:348] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.419589 12703 raft_consensus.cc:374] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:41:31.419683 12703 raft_consensus.cc:729] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a617794e7ac4ed884baa7f785c2145c, State: Initialized, Role: FOLLOWER
I20260927 21:41:31.419582 12702 raft_consensus.cc:348] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.419997 12702 raft_consensus.cc:374] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:41:31.420099 12702 raft_consensus.cc:729] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489c0ae064364e70bfa30a2324891d96, State: Initialized, Role: FOLLOWER
I20260927 21:41:31.420516 12703 consensus_queue.cc:260] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.420713 12702 consensus_queue.cc:260] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.421689 12701 raft_consensus.cc:348] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.422254 12701 raft_consensus.cc:374] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:41:31.422374 12701 raft_consensus.cc:729] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc3582e23914c8d9d7f6300caecc64a, State: Initialized, Role: FOLLOWER
I20260927 21:41:31.423087 12701 consensus_queue.cc:260] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.423823 12703 ts_tablet_manager.cc:1428] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20260927 21:41:31.424793 12548 heartbeater.cc:503] Master 127.0.202.190:46067 was elected leader, sending a full tablet report...
I20260927 21:41:31.425395 12702 ts_tablet_manager.cc:1428] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260927 21:41:31.426188 12682 heartbeater.cc:503] Master 127.0.202.190:46067 was elected leader, sending a full tablet report...
I20260927 21:41:31.428005 12701 ts_tablet_manager.cc:1428] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260927 21:41:31.501361 12683 tablet.cc:2369] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:41:31.526307 12406 tablet.cc:2369] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:41:31.533972 12707 raft_consensus.cc:482] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:41:31.534328 12707 raft_consensus.cc:504] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.536196 12707 leader_election.cc:288] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc3582e23914c8d9d7f6300caecc64a (127.0.202.131:44989), 489c0ae064364e70bfa30a2324891d96 (127.0.202.130:42705)
I20260927 21:41:31.544188 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.536744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:32853 (local address 127.0.202.130:42705)
0927 21:41:31.536962 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.536969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.537863 (+ 894us) server_negotiation.cc:407] Connection header received
0927 21:41:31.538177 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.538182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.538253 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.538355 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:41:31.539045 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.539828 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.542318 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.543087 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.543238 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.543323 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.543759 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.543896 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:41:31.544214 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.536650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:45335 (local address 127.0.202.131:44989)
0927 21:41:31.536862 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.536870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.538000 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 21:41:31.538181 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.538185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.538252 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.538356 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:41:31.538979 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.539719 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.542199 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.543111 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.543332 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.543411 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.543757 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.543908 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:41:31.544456 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.536746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:42705 (local address 127.0.202.129:32853)
0927 21:41:31.537737 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:31.537757 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:31.537937 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:31.538371 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:31.538375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:31.538483 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:41:31.538871 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.538880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.540014 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.540019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.542145 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.542162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.543335 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.543342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.543567 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.543614 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:41:31.543728 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:41:31.543947 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":66,"spinlock_wait_cycles":28544,"thread_start_us":553,"threads_started":1}
I20260927 21:41:31.544728 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.537002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:44989 (local address 127.0.202.129:45335)
0927 21:41:31.537882 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 21:41:31.537897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:41:31.538020 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:41:31.538371 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:41:31.538376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:41:31.538400 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:41:31.538789 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.538798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.539849 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.539855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.542019 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:41:31.542035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.543386 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.543391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:41:31.543568 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.543617 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:41:31.543726 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:41:31.543906 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":160,"thread_start_us":559,"threads_started":1}
I20260927 21:41:31.545645 12503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0680324534e04ae82e1504a6222a0" candidate_uuid: "0a617794e7ac4ed884baa7f785c2145c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "489c0ae064364e70bfa30a2324891d96" is_pre_election: true
I20260927 21:41:31.545943 12637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0680324534e04ae82e1504a6222a0" candidate_uuid: "0a617794e7ac4ed884baa7f785c2145c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc3582e23914c8d9d7f6300caecc64a" is_pre_election: true
I20260927 21:41:31.546202 12503 raft_consensus.cc:2454] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a617794e7ac4ed884baa7f785c2145c in term 0.
I20260927 21:41:31.546509 12637 raft_consensus.cc:2454] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a617794e7ac4ed884baa7f785c2145c in term 0.
I20260927 21:41:31.546984 12296 leader_election.cc:302] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a617794e7ac4ed884baa7f785c2145c, 489c0ae064364e70bfa30a2324891d96; no voters:
I20260927 21:41:31.547505 12707 raft_consensus.cc:2781] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:41:31.547653 12707 raft_consensus.cc:482] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:41:31.547766 12707 raft_consensus.cc:3037] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:41:31.550680 12707 raft_consensus.cc:504] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.551477 12707 leader_election.cc:288] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [CANDIDATE]: Term 1 election: Requested vote from peers 5dc3582e23914c8d9d7f6300caecc64a (127.0.202.131:44989), 489c0ae064364e70bfa30a2324891d96 (127.0.202.130:42705)
I20260927 21:41:31.552014 12637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0680324534e04ae82e1504a6222a0" candidate_uuid: "0a617794e7ac4ed884baa7f785c2145c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc3582e23914c8d9d7f6300caecc64a"
I20260927 21:41:31.552084 12503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0680324534e04ae82e1504a6222a0" candidate_uuid: "0a617794e7ac4ed884baa7f785c2145c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "489c0ae064364e70bfa30a2324891d96"
I20260927 21:41:31.552266 12637 raft_consensus.cc:3037] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:41:31.552383 12503 raft_consensus.cc:3037] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:41:31.554956 12503 raft_consensus.cc:2454] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a617794e7ac4ed884baa7f785c2145c in term 1.
I20260927 21:41:31.555346 12637 raft_consensus.cc:2454] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a617794e7ac4ed884baa7f785c2145c in term 1.
I20260927 21:41:31.555408 12296 leader_election.cc:302] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a617794e7ac4ed884baa7f785c2145c, 489c0ae064364e70bfa30a2324891d96; no voters:
I20260927 21:41:31.555760 12707 raft_consensus.cc:2781] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:41:31.556345 12707 raft_consensus.cc:686] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 1 LEADER]: Becoming Leader. State: Replica: 0a617794e7ac4ed884baa7f785c2145c, State: Running, Role: LEADER
I20260927 21:41:31.556931 12707 consensus_queue.cc:237] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } }
I20260927 21:41:31.561504 12190 catalog_manager.cc:5495] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a617794e7ac4ed884baa7f785c2145c (127.0.202.129). New cstate: current_term: 1 leader_uuid: "0a617794e7ac4ed884baa7f785c2145c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a617794e7ac4ed884baa7f785c2145c" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 } health_report { overall_health: UNKNOWN } } }
I20260927 21:41:31.606277 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.600468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.0.202.129:37045)
0927 21:41:31.600692 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.600698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.600745 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:41:31.601059 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.601063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.601158 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.601322 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:41:31.601727 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.602473 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.604008 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.604506 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.604644 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.605175 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:41:31.605693 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:41:31.605861 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.605913 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.605995 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:41:31.615769 12637 raft_consensus.cc:1261] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 1 FOLLOWER]: Refusing update from remote peer 0a617794e7ac4ed884baa7f785c2145c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:41:31.615769 12503 raft_consensus.cc:1261] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 1 FOLLOWER]: Refusing update from remote peer 0a617794e7ac4ed884baa7f785c2145c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:41:31.617125 12707 consensus_queue.cc:1035] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [LEADER]: Connected to new peer: Peer: permanent_uuid: "489c0ae064364e70bfa30a2324891d96" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:41:31.617542 12712 consensus_queue.cc:1035] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc3582e23914c8d9d7f6300caecc64a" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:41:31.632823 12715 mvcc.cc:205] Tried to move back new op lower bound from 7334073514446872576 to 7334073514220974080. Current Snapshot: MvccSnapshot[applied={T|T < 7334073514446872576}]
I20260927 21:41:31.637038 12717 mvcc.cc:205] Tried to move back new op lower bound from 7334073514446872576 to 7334073514220974080. Current Snapshot: MvccSnapshot[applied={T|T < 7334073514446872576}]
I20260927 21:41:31.662216 12191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:41:31.662503 12191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:41:31.666194 12191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cab4dfe0a424c308dadf5b311c5f6a7: Sending DeleteTablet for 3 replicas of tablet 0cb0680324534e04ae82e1504a6222a0
I20260927 21:41:31.667493 12617 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cb0680324534e04ae82e1504a6222a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:41:31 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:51480
I20260927 21:41:31.667438 12483 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cb0680324534e04ae82e1504a6222a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:41:31 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:55944
I20260927 21:41:31.667569 12340 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cb0680324534e04ae82e1504a6222a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:41:31 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:36804
I20260927 21:41:31.672787 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 12272
I20260927 21:41:31.680150 12726 tablet_replica.cc:330] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: stopping tablet replica
I20260927 21:41:31.680652 12727 tablet_replica.cc:330] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: stopping tablet replica
I20260927 21:41:31.680862 12726 raft_consensus.cc:2229] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:41:31.681129 12727 raft_consensus.cc:2229] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:41:31.681470 12726 raft_consensus.cc:2258] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:41:31.681948 12727 raft_consensus.cc:2258] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:41:31.684065 12726 ts_tablet_manager.cc:1905] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:41:31.684266 12727 ts_tablet_manager.cc:1905] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:41:31.685889 12725 tablet_replica.cc:330] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: stopping tablet replica
I20260927 21:41:31.686410 12725 raft_consensus.cc:2229] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:41:31.687022 12725 raft_consensus.cc:2258] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:41:31.690757 12725 ts_tablet_manager.cc:1905] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:41:31.693508 12727 ts_tablet_manager.cc:1918] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:41:31.693702 12727 log.cc:1196] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/0cb0680324534e04ae82e1504a6222a0
I20260927 21:41:31.694311 12727 ts_tablet_manager.cc:1939] T 0cb0680324534e04ae82e1504a6222a0 P 0a617794e7ac4ed884baa7f785c2145c: Deleting consensus metadata
I20260927 21:41:31.696516 12179 catalog_manager.cc:4841] TS 0a617794e7ac4ed884baa7f785c2145c (127.0.202.129:37045): tablet 0cb0680324534e04ae82e1504a6222a0 (table test-table [id=2ba740ff982f40b69629a7c2f76d4789]) successfully deleted
I20260927 21:41:31.697875 12725 ts_tablet_manager.cc:1918] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:41:31.698040 12725 log.cc:1196] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/0cb0680324534e04ae82e1504a6222a0
I20260927 21:41:31.698599 12725 ts_tablet_manager.cc:1939] T 0cb0680324534e04ae82e1504a6222a0 P 489c0ae064364e70bfa30a2324891d96: Deleting consensus metadata
I20260927 21:41:31.700548 12179 catalog_manager.cc:4841] TS 489c0ae064364e70bfa30a2324891d96 (127.0.202.130:42705): tablet 0cb0680324534e04ae82e1504a6222a0 (table test-table [id=2ba740ff982f40b69629a7c2f76d4789]) successfully deleted
Sep 27 21:41:31 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:41:31.703691 12726 ts_tablet_manager.cc:1918] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:41:31.703873 12726 log.cc:1196] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/0cb0680324534e04ae82e1504a6222a0
I20260927 21:41:31.704468 12726 ts_tablet_manager.cc:1939] T 0cb0680324534e04ae82e1504a6222a0 P 5dc3582e23914c8d9d7f6300caecc64a: Deleting consensus metadata
I20260927 21:41:31.706512 12177 catalog_manager.cc:4841] TS 5dc3582e23914c8d9d7f6300caecc64a (127.0.202.131:44989): tablet 0cb0680324534e04ae82e1504a6222a0 (table test-table [id=2ba740ff982f40b69629a7c2f76d4789]) successfully deleted
I20260927 21:41:31.709474 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.675756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.202.129:37045)
0927 21:41:31.676417 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.676424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.689952 (+ 13528us) server_negotiation.cc:407] Connection header received
0927 21:41:31.690489 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.690494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.690582 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.690653 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:31.692596 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.693900 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.696207 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.696912 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.697035 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.702906 (+ 5871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:31.703015 (+ 109us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:31.703032 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:31.703081 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:31.706371 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.707196 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.707204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.707206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.707356 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.707751 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.707761 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.707763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.708154 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:31.708500 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.708857 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.709131 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467}
I20260927 21:41:31.710143 12400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:41:31.814625 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 12272
I20260927 21:41:31.830216 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 12411
Sep 27 21:41:31 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:41:31.849066 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.831171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.0.202.130:42705)
0927 21:41:31.831383 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.831390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.831410 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:41:31.831566 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.831571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.831688 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.831795 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:31.833343 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.834214 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.835562 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.836024 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.836145 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.841938 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:31.841964 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:31.841987 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:31.842041 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:31.845790 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.846585 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.846594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.846596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.846741 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.847144 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.847148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.847150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.847643 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:31.847947 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.848275 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.848567 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:41:31.849943 12543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:41:31.941831 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 12411
I20260927 21:41:31.958791 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 12554
Sep 27 21:41:31 dist-test-slave-25dq krb5kdc[11716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:41:31.975167 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:41:31.959705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.0.202.131:44989)
0927 21:41:31.959888 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:41:31.959895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:41:31.959938 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:41:31.960190 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:41:31.960194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:41:31.960266 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:41:31.960432 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:41:31.961787 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.962553 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.963829 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:41:31.964225 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:41:31.964532 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:41:31.968924 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:41:31.968940 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:41:31.968956 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:41:31.968992 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:41:31.972177 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.973035 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.973045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.973049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.973187 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:41:31.973584 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:41:31.973590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:41:31.973593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:41:31.973967 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:41:31.974278 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 21:41:31.974585 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:41:31.974815 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:41:31.975894 12677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:41:32.070909 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 12554
I20260927 21:41:32.088940 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 12160
I20260927 21:41:32.090267 12222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:41:32.205660 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 12160
I20260927 21:41:32.230325 810 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:41:32.566452 810 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:41:32.566735 810 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:41:32.912941 810 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:41:32.913 UTC [11746] LOG: received smart shutdown request
2026-09-27 21:41:32.917 UTC [11746] LOG: background worker "logical replication launcher" (PID 11755) exited with exit code 1
2026-09-27 21:41:32.918 UTC [11750] LOG: shutting down
2026-09-27 21:41:33.134 UTC [11746] LOG: database system is shut down
2026-09-27T21:41:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52875 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:41:33 dist-test-slave-25dq krb5kdc[12741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:41:33 dist-test-slave-25dq krb5kdc[12741](info): set up 2 sockets
Sep 27 21:41:33 dist-test-slave-25dq krb5kdc[12741](info): commencing operation
krb5kdc: starting...
W20260927 21:41:34.393033 810 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:41:34 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:41:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/rangeradmin_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/rangeradmin_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/rangerlookup_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/rangerlookup_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
I20260927 21:41:34.739426 810 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3Z3URy/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/postgres -l logfile start
2026-09-27 21:41:37.644 UTC [12771] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:41:37.644 UTC [12771] LOG: listening on IPv4 address "127.0.202.148", port 36393
2026-09-27 21:41:37.648 UTC [12771] LOG: listening on Unix socket "/tmp/.s.PGSQL.36393"
2026-09-27 21:41:37.671 UTC [12774] LOG: database system was shut down at 2026-09-27 21:41:35 UTC
2026-09-27 21:41:37.676 UTC [12771] LOG: database system is ready to accept connections
I20260927 21:41:38.708590 810 mini_postgres.cc:96] Postgres bound to 36393
2026-09-27 21:41:38.726 UTC [12782] FATAL: database "slave" does not exist
127.0.202.148:36393 - accepting connections
I20260927 21:41:38.727010 810 mini_ranger.cc:161] Starting Ranger...
I20260927 21:41:38.756270 810 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:41:39.374698 810 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:41:39.374831 810 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-admin
2026-09-27 21:41:39,671 [I] DB FLAVOR :POSTGRES
2026-09-27 21:41:39,672 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:41:39,672 [I] Checking connection..
2026-09-27 21:41:39,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:41:39,945 [I] Checking connection passed.
2026-09-27 21:41:39,945 [I] --------- Verifying version history table ---------
2026-09-27 21:41:39,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:41:40,201 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:41:40,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:41:40,472 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:41:40,472 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:41:40,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:41:40.731 UTC [12846] WARNING: there is no transaction in progress
2026-09-27 21:41:40,738 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:41:40,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:41:40,996 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:41:40,997 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:41:40,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:41:41,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:41:41,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-25dq.c.gcp-upstream.internal','N') ;"
2026-09-27 21:41:41,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:41:42,022 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:41:42,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:41:42,271 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:41:42,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:41:42,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:41:42,782 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:41:42,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:41:43.868 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:43.887 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:43.908 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:43.927 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:43.946 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:43.965 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.026 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.034 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.050 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.062 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.074 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.089 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.099 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.106 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.114 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44.122 UTC [12954] WARNING: there is no transaction in progress
2026-09-27 21:41:44,923 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:41:44,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:41:45,187 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:41:45,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:41:45,443 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:41:45,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:41:45,696 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:41:45,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:41:45,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:41:46,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-25dq.c.gcp-upstream.internal';"
2026-09-27 21:41:46,488 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:41:46,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:41:46,736 [I] DB_PATCHES have already been applied
I20260927 21:41:46.743237 810 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-admin:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3Z3URy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:41:46.743347 810 mini_ranger.cc:193] Using host: 127.0.202.148
I20260927 21:41:46.758999 810 mini_ranger.cc:227] Ranger admin URL: http://127.0.202.148:59467
Sep 27, 2026 9:41:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:41:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:41:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:41:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:41:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:41:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:41:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:41:47.971170 810 mini_ranger.cc:160] Time spent starting Ranger: real 9.244s user 0.002s sys 0.042s
Sep 27, 2026 9:42:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:42:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 9:42:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:42:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:42:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:42:11.058930 810 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/rangerkms_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/rangerkms_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.202.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.202.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.202.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/HTTP_127.0.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:42:11 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545331, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:42:11.231988 810 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:42:11.254247 810 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:42:12.322156 810 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:42:12.322283 810 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms
I20260927 21:42:12.322337 810 mini_ranger_kms.cc:227] Using postgres at 127.0.202.148:36393
2026-09-27 21:42:12,696 [I] DB FLAVOR :POSTGRES
2026-09-27 21:42:12,697 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:42:12,697 [I] Checking connection
2026-09-27 21:42:12,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:42:12,962 [I] connection success
2026-09-27 21:42:12,962 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:42:12,962 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:42:12,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:42:13,200 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:42:13,200 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:42:13,200 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:42:13,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.202.148:36393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:42:13,491 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:42:14.034415 810 mini_ranger_kms.cc:310] Created kms service
I20260927 21:42:14.215010 810 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:42:14.266134 810 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:42:14 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545331, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:14.695387 810 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:42:14.695703 810 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms:/tmp/dist-test-task3Z3URy/build/asan/bin/postgresql.jar:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3Z3URy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:42:14.695755 810 mini_ranger_kms.cc:240] Using host: 127.0.202.148
I20260927 21:42:14.716564 810 mini_ranger_kms.cc:276] Ranger KMS PID: 13135
I20260927 21:42:14.716714 810 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.202.148:46365
Sep 27, 2026 9:42:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:42:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:42:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:42:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:42:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:42:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 21:42:16 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790545336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.202.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:17.114055 810 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.882s user 0.070s sys 0.321s
I20260927 21:42:17.114315 810 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:42:17.114477 810 mini_ranger_kms.cc:401] 127.0.202.148:46365/kms/v1/keys
Sep 27 21:42:17 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.202.148@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
Sep 27 21:42:17 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.202.148@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
Sep 27 21:42:20 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:42:20.113693 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41403
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:41403
--ranger_config_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:20.456166 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:20.456609 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:20.456771 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:20.464454 13185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:20.464604 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:20.464711 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:20.464802 13185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:20.464874 13185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:20.475435 13185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.202.148:46365/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:41403
--ranger_config_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:20.476953 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:20.478331 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:20.490552 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:20.490741 13193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:20.491103 13190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:20.492408 13185 server_base.cc:1034] running on GCE node
I20260927 21:42:20.493427 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:20.495410 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:20.496660 13185 hybrid_clock.cc:648] HybridClock initialized: now 1790545340496589 us; error 67 us; skew 500 ppm
I20260927 21:42:20.497115 13185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:20 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:20.508726 13185 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:42:20.510802 13185 webserver.cc:460] Webserver started at http://127.0.202.190:46195/ using document root <none> and password file <none>
I20260927 21:42:20.511612 13185 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:20.511777 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:20.512149 13185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:42:20 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:20.579409 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "fed70ae400274a798c6fe0e68ff047e4"
format_stamp: "Formatted at 2026-09-27 21:42:20 on dist-test-slave-25dq"
server_key: "37150a96db567ec6e7078b2eaff22439"
server_key_iv: "d636179da765a0c1134ead0a8c8cd35f"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:20.580575 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "fed70ae400274a798c6fe0e68ff047e4"
format_stamp: "Formatted at 2026-09-27 21:42:20 on dist-test-slave-25dq"
server_key: "37150a96db567ec6e7078b2eaff22439"
server_key_iv: "d636179da765a0c1134ead0a8c8cd35f"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:20.586720 13185 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:42:20.607828 13200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:20.609517 13185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:42:20.609704 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "fed70ae400274a798c6fe0e68ff047e4"
format_stamp: "Formatted at 2026-09-27 21:42:20 on dist-test-slave-25dq"
server_key: "37150a96db567ec6e7078b2eaff22439"
server_key_iv: "d636179da765a0c1134ead0a8c8cd35f"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:20.609982 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:20.640250 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:20.644662 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:20.644990 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:20.665194 13185 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:41403
I20260927 21:42:20.665197 13251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:41403 every 8 connection(s)
I20260927 21:42:20.666599 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:42:20.670619 13252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:20.671885 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 13185
I20260927 21:42:20.672091 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:42:20.672911 810 external_mini_cluster.cc:1428] Setting key 1d3f20bcf17c54eccd2da10485d80e13
I20260927 21:42:20.688601 13252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Bootstrap starting.
I20260927 21:42:20.693183 13252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:20.694521 13252 log.cc:824] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:20.697958 13252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: No bootstrap required, opened a new log
Sep 27 21:42:20 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:20.709189 13252 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:20.709672 13252 raft_consensus.cc:374] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:20.709841 13252 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed70ae400274a798c6fe0e68ff047e4, State: Initialized, Role: FOLLOWER
I20260927 21:42:20.710708 13252 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:20.711054 13252 raft_consensus.cc:388] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:42:20.711225 13252 raft_consensus.cc:482] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:42:20.711418 13252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:20.712706 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:20.674977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.0.202.190:41403)
0927 21:42:20.690157 (+ 15180us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:20.690172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:20.690223 (+ 51us) server_negotiation.cc:407] Connection header received
0927 21:42:20.692786 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:20.692816 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:20.693378 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:20.693892 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:20.695482 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:20.697019 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:20.698491 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:20.699154 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:20.699495 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:20.703822 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:20.703831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:20.703854 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:20.703924 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:20.707447 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:20.708212 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:20.708227 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:20.708234 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:20.708458 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:20.708883 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:20.708891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:20.708894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:20.709568 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:20.709899 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:20.710747 (+ 848us) server_negotiation.cc:299] Negotiation successful
0927 21:42:20.711436 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14927,"thread_start_us":14692,"threads_started":1}
I20260927 21:42:20.714824 13252 raft_consensus.cc:504] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:20.715447 13252 leader_election.cc:302] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fed70ae400274a798c6fe0e68ff047e4; no voters:
I20260927 21:42:20.716794 13252 leader_election.cc:288] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:42:20.716856 13257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:42:20.719129 13252 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:20.723927 13257 raft_consensus.cc:686] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 LEADER]: Becoming Leader. State: Replica: fed70ae400274a798c6fe0e68ff047e4, State: Running, Role: LEADER
I20260927 21:42:20.724655 13257 consensus_queue.cc:237] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:20.727772 13258 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fed70ae400274a798c6fe0e68ff047e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } } }
I20260927 21:42:20.728194 13258 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:20.728735 13259 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fed70ae400274a798c6fe0e68ff047e4. Latest consensus state: current_term: 1 leader_uuid: "fed70ae400274a798c6fe0e68ff047e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } } }
I20260927 21:42:20.729028 13259 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:20.751216 13265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:42:20.764499 13265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:42:20.779178 13265 catalog_manager.cc:1347] Generated new cluster ID: 95cef6d27f5e4e63960239996347b8db
I20260927 21:42:20.779429 13265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:42:20.781394 13252 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:42:20.822688 13265 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:42:20.824950 13265 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:42:20.843243 13265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Generated new TSK 0
I20260927 21:42:20.844568 13265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:42:21.685406 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41403
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:42:22 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790545342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:42:22.090723 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:22.091101 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:22.091315 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:22.099900 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:22.100050 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:22.100238 13298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:42:22.111034 13298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.202.148:46365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:22.112452 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:22.114224 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:22.131666 13308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:22.133294 13309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:22.136440 13298 server_base.cc:1034] running on GCE node
W20260927 21:42:22.136929 13311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:22.137693 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:22.139420 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:22.140692 13298 hybrid_clock.cc:648] HybridClock initialized: now 1790545342140624 us; error 73 us; skew 500 ppm
I20260927 21:42:22.141145 13298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:22 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:22.145603 13298 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:42:22.147890 13298 webserver.cc:460] Webserver started at http://127.0.202.129:35335/ using document root <none> and password file <none>
I20260927 21:42:22.148883 13298 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:22.149034 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:22.149392 13298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:42:22 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:22.177991 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "9129963a46664aeeb00fa9f6f16c46f8"
format_stamp: "Formatted at 2026-09-27 21:42:22 on dist-test-slave-25dq"
server_key: "6fc136a1fe23a60909c9542d25fe6b23"
server_key_iv: "5c651d07431d5ed7d1da066950a73a15"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:22.178943 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "9129963a46664aeeb00fa9f6f16c46f8"
format_stamp: "Formatted at 2026-09-27 21:42:22 on dist-test-slave-25dq"
server_key: "6fc136a1fe23a60909c9542d25fe6b23"
server_key_iv: "5c651d07431d5ed7d1da066950a73a15"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:22.190064 13298 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.008s
I20260927 21:42:22.224982 13318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:22.226927 13298 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.004s sys 0.000s
I20260927 21:42:22.227151 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "9129963a46664aeeb00fa9f6f16c46f8"
format_stamp: "Formatted at 2026-09-27 21:42:22 on dist-test-slave-25dq"
server_key: "6fc136a1fe23a60909c9542d25fe6b23"
server_key_iv: "5c651d07431d5ed7d1da066950a73a15"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:22.227391 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:22.258253 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:22.262796 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:22.263145 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:22.264482 13298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:22.266068 13298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:22.266170 13298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:22.266300 13298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:22.266387 13298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:22.349933 13298 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:46475
I20260927 21:42:22.349952 13430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:46475 every 8 connection(s)
I20260927 21:42:22.351627 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:42:22.359329 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 13298
I20260927 21:42:22.359579 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:42:22.360265 810 external_mini_cluster.cc:1428] Setting key 45eb1c8bd4098c2323e37e070fd44109
Sep 27 21:42:22 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:22.392648 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:22.355455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41403 (local address 127.0.202.129:53223)
0927 21:42:22.356419 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:22.356472 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:22.358367 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:22.359344 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:22.359360 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:22.360090 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:22.361158 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:22.361199 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:22.362814 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:22.362820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:22.364554 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:22.364580 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:22.369719 (+ 5139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:22.369728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:22.370089 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:22.378507 (+ 8418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:22.378555 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:22.382312 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:22.387129 (+ 4817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:22.387141 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:22.387164 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:22.387860 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:22.388428 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:22.388434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:22.388438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:22.388749 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:22.389549 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:22.389559 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:22.390079 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 21:42:22.390437 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 21:42:22.391075 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":405,"threads_started":1}
I20260927 21:42:22.393931 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:22.354809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:53223 (local address 127.0.202.190:41403)
0927 21:42:22.355348 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:22.355353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:22.356604 (+ 1251us) server_negotiation.cc:407] Connection header received
0927 21:42:22.358699 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:22.358705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:22.358818 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:22.358974 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:22.361383 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:22.362631 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:22.368443 (+ 5812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:22.369215 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:22.369365 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:22.382708 (+ 13343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:22.382717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:22.382723 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:22.382790 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:22.386875 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:22.388064 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:22.388071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:22.388076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:22.388256 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:22.388900 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:22.388906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:22.388911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:22.389302 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:22.389443 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:22.393259 (+ 3816us) server_negotiation.cc:299] Negotiation successful
0927 21:42:22.393520 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":311,"threads_started":1}
I20260927 21:42:22.396375 13431 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41403
I20260927 21:42:22.396646 13431 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:22.397574 13431 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
I20260927 21:42:22.401690 13217 ts_manager.cc:194] Registered new tserver with Master: 9129963a46664aeeb00fa9f6f16c46f8 (127.0.202.129:46475)
I20260927 21:42:22.403949 13217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:53223
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:42:22.449463 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41403
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:42:22 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
Sep 27 21:42:22 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
W20260927 21:42:22.951421 13437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:22.951936 13437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:22.952121 13437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:22.963703 13437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:22.963930 13437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:22.964146 13437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:42:22.980046 13437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.202.148:46365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:22.982041 13437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:22.983847 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:23.041937 13444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:23.043427 13445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:23.064867 13447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:23.070281 13437 server_base.cc:1034] running on GCE node
I20260927 21:42:23.071249 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:23.072827 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:23.080407 13437 hybrid_clock.cc:648] HybridClock initialized: now 1790545343080275 us; error 109 us; skew 500 ppm
I20260927 21:42:23.080935 13437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:23 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:23.085542 13437 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:42:23.116904 13437 webserver.cc:460] Webserver started at http://127.0.202.130:40611/ using document root <none> and password file <none>
I20260927 21:42:23.118148 13437 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:23.118386 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:23.118871 13437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:42:23 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:23.142870 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "be0eece6578b49818b9d4313d9b537cb"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "3f33117eb0faa830cc6db1878f7f4029"
server_key_iv: "6890f48309db5a9ee23a1c9f89ab351c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:23.144506 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "be0eece6578b49818b9d4313d9b537cb"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "3f33117eb0faa830cc6db1878f7f4029"
server_key_iv: "6890f48309db5a9ee23a1c9f89ab351c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:23.170506 13437 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.007s sys 0.016s
I20260927 21:42:23.195026 13461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:23.197328 13437 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.008s
I20260927 21:42:23.197633 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "be0eece6578b49818b9d4313d9b537cb"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "3f33117eb0faa830cc6db1878f7f4029"
server_key_iv: "6890f48309db5a9ee23a1c9f89ab351c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:23.197993 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:23.235512 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:23.241206 13437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:23.241608 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:23.242918 13437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:23.244880 13437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:23.245019 13437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:42:23.245167 13437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:23.245251 13437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:23.289018 13437 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:40775
I20260927 21:42:23.289048 13573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:40775 every 8 connection(s)
I20260927 21:42:23.290748 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:42:23.299614 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 13437
I20260927 21:42:23.299784 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:42:23.300273 810 external_mini_cluster.cc:1428] Setting key 15193b549ad0821ae6479bada5556a03
Sep 27 21:42:23 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:42:23.335338 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:23.294931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:55305 (local address 127.0.202.190:41403)
0927 21:42:23.304451 (+ 9520us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:23.304459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:23.304481 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:42:23.304542 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:23.304550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:23.304649 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:23.304759 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:23.315413 (+ 10654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.316705 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.319276 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.319904 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.320009 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:23.324439 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:23.324449 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:23.324457 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:23.324505 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:23.327974 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:23.329100 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:23.329107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:23.329111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:23.329273 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:23.332979 (+ 3706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:23.332985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:23.332989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:23.333392 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:23.333561 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:23.334697 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 21:42:23.334961 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9378,"thread_start_us":14410,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:42:23.336884 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:23.294321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41403 (local address 127.0.202.130:55305)
0927 21:42:23.300970 (+ 6649us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:23.301045 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:23.303642 (+ 2597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:23.313353 (+ 9711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:23.313370 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:23.314119 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:23.315125 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:23.315160 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.316957 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.316963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:23.319076 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:23.319095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.320384 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.320392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:23.320676 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:23.321507 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:23.321538 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:23.324105 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:23.328199 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:23.328210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:23.328231 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:23.328896 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:23.332423 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:23.332429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:23.332433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:23.332790 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:23.333662 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:23.333671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:23.334157 (+ 486us) client_negotiation.cc:769] Sending connection context
0927 21:42:23.334546 (+ 389us) client_negotiation.cc:240] Negotiation successful
0927 21:42:23.335156 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6275,"thread_start_us":14011,"threads_started":1}
I20260927 21:42:23.339159 13574 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41403
I20260927 21:42:23.339509 13574 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:23.340677 13574 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
I20260927 21:42:23.342514 13217 ts_manager.cc:194] Registered new tserver with Master: be0eece6578b49818b9d4313d9b537cb (127.0.202.130:40775)
I20260927 21:42:23.343652 13217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:55305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:42:23.372780 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41403
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:23.408950 13431 heartbeater.cc:503] Master 127.0.202.190:41403 was elected leader, sending a full tablet report...
W20260927 21:42:23.717996 13580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:23.718333 13580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:23.718490 13580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:23.726212 13580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:23.726364 13580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:23.726533 13580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:42:23.736569 13580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.202.148:46365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:23.737892 13580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:23.739202 13580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:23.752915 13585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:23.752988 13586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:23.753156 13588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:23.757901 13580 server_base.cc:1034] running on GCE node
I20260927 21:42:23.758641 13580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:23.759709 13580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:23.761011 13580 hybrid_clock.cc:648] HybridClock initialized: now 1790545343760963 us; error 113 us; skew 500 ppm
I20260927 21:42:23.761404 13580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:23 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:23.765090 13580 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:42:23.766841 13580 webserver.cc:460] Webserver started at http://127.0.202.131:39855/ using document root <none> and password file <none>
I20260927 21:42:23.767609 13580 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:23.767730 13580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:23.768131 13580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:42:23 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:23.783783 13580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "36dadcbcb66d4e0496c674bc2b9a059b"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "8e428c29899fc4e56419ce9e32a9f688"
server_key_iv: "901e486df3bd070b8629721cb397779c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:23.784741 13580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "36dadcbcb66d4e0496c674bc2b9a059b"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "8e428c29899fc4e56419ce9e32a9f688"
server_key_iv: "901e486df3bd070b8629721cb397779c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:23.791056 13580 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:42:23.806232 13595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:23.807909 13580 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 21:42:23.808094 13580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "36dadcbcb66d4e0496c674bc2b9a059b"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "8e428c29899fc4e56419ce9e32a9f688"
server_key_iv: "901e486df3bd070b8629721cb397779c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:23.808380 13580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:23.839639 13580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:23.844486 13580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:23.844827 13580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:23.845849 13580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:23.847383 13580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:23.847462 13580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:23.847637 13580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:23.847759 13580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:23.885725 13580 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37149
I20260927 21:42:23.885761 13707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37149 every 8 connection(s)
I20260927 21:42:23.887036 13580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:42:23.891820 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 13580
I20260927 21:42:23.892002 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:42:23.892534 810 external_mini_cluster.cc:1428] Setting key a468a603a3b5eecf4e33e4b41883dca2
Sep 27 21:42:23 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:23.907847 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:23.889524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:53253 (local address 127.0.202.190:41403)
0927 21:42:23.889970 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:23.889975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:23.891482 (+ 1507us) server_negotiation.cc:407] Connection header received
0927 21:42:23.893610 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:23.893614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:23.893693 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:23.893786 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:23.895651 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.896461 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.898019 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.898451 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.898582 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:23.901867 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:23.901873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:23.901878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:23.901921 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:23.904639 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:23.905440 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:23.905452 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:23.905456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:23.905589 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:23.906017 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:23.906024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:23.906029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:23.906307 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:23.906461 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:23.907242 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 21:42:23.907459 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260927 21:42:23.908890 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:23.890189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41403 (local address 127.0.202.131:53253)
0927 21:42:23.891321 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:23.891359 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:23.893344 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:23.894098 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:23.894106 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:23.894637 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:23.895475 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:23.895500 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.896611 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.896616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:23.897857 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:23.897876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.898663 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.898670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:23.898908 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:23.899602 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:23.899624 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:23.901643 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:23.904774 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:23.904783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:23.904801 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:23.905317 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:23.905702 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:23.905708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:23.905711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:23.905907 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:23.906431 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:23.906440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:23.906838 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 21:42:23.907151 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 21:42:23.907658 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":671,"threads_started":1}
I20260927 21:42:23.910243 13708 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41403
I20260927 21:42:23.910560 13708 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:23.911358 13708 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
I20260927 21:42:23.912850 13217 ts_manager.cc:194] Registered new tserver with Master: 36dadcbcb66d4e0496c674bc2b9a059b (127.0.202.131:37149)
I20260927 21:42:23.913899 13217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:53253
I20260927 21:42:23.921010 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:42:23.922793 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 13298
Sep 27 21:42:23 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:42:23.940461 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:23.923906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.0.202.129:46475)
0927 21:42:23.924606 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:23.924613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:23.924638 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:42:23.924693 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:23.924699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:23.924901 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:23.925031 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:23.926749 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.927789 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.929355 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:23.929907 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:23.930007 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:23.934350 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:23.934359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:23.934372 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:23.934406 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:23.937415 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:23.938190 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:23.938199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:23.938203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:23.938401 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:23.938778 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:23.938783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:23.938785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:23.939178 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:23.939428 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:23.939761 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 21:42:23.939966 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":371,"threads_started":1}
I20260927 21:42:23.941717 13426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:24.029168 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 13298
I20260927 21:42:24.045102 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 13437
Sep 27 21:42:24 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:42:24.062788 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:24.046017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.202.130:40775)
0927 21:42:24.046593 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:24.046600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:24.046626 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:42:24.046725 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:24.046732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:24.046950 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:24.047157 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:24.048357 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:24.049350 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:24.051365 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:24.052170 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:24.052362 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:24.056885 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:24.056897 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:24.056912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:24.056954 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:24.060059 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:24.060838 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:24.060846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:24.060848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:24.061003 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:24.061334 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:24.061338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:24.061340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:24.061715 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:24.061942 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:24.062171 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 21:42:24.062371 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":300,"threads_started":1}
I20260927 21:42:24.064177 13569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:24.147840 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 13437
I20260927 21:42:24.167801 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 13580
Sep 27 21:42:24 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:42:24.186268 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:24.168976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.0.202.131:37149)
0927 21:42:24.169542 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:24.169550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:24.169577 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:42:24.169636 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:24.169642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:24.169859 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:24.170010 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:24.171223 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:24.172252 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:24.173937 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:24.174519 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:24.174621 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:24.179936 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:24.179948 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:24.179963 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:24.180000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:24.182787 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:24.183590 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:24.183596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:24.183599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:24.183719 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:24.184197 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:24.184201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:24.184204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:24.184972 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:24.185276 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:24.185568 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:42:24.185788 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":266,"threads_started":1}
I20260927 21:42:24.187711 13703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:24.270252 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 13580
I20260927 21:42:24.286042 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 13185
I20260927 21:42:24.287530 13247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:24.395702 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 13185
I20260927 21:42:24.420255 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41403
--webserver_interface=127.0.202.190
--webserver_port=46195
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:41403
--ranger_config_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:24.757722 13720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:24.758041 13720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:24.758184 13720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:24.765719 13720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:24.765832 13720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:24.765939 13720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:24.766031 13720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:24.766115 13720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:24.776435 13720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:41403
--ranger_config_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=46195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:24.777784 13720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:24.779173 13720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:24.791602 13726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:24.791715 13725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:24.791693 13728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:24.792449 13720 server_base.cc:1034] running on GCE node
I20260927 21:42:24.793200 13720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:24.794750 13720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:24.795987 13720 hybrid_clock.cc:648] HybridClock initialized: now 1790545344795909 us; error 74 us; skew 500 ppm
I20260927 21:42:24.796555 13720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:24 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:24.799943 13720 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:42:24.801772 13720 webserver.cc:460] Webserver started at http://127.0.202.190:46195/ using document root <none> and password file <none>
I20260927 21:42:24.802525 13720 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:24.802680 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:42:24 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:24.854948 13720 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:42:24.858995 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:24.860877 13720 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 21:42:24.861111 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "fed70ae400274a798c6fe0e68ff047e4"
format_stamp: "Formatted at 2026-09-27 21:42:20 on dist-test-slave-25dq"
server_key: "37150a96db567ec6e7078b2eaff22439"
server_key_iv: "d636179da765a0c1134ead0a8c8cd35f"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:24.862231 13720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:24.887671 13720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:24.891940 13720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:24.892268 13720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:24.912997 13720 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:41403
I20260927 21:42:24.913022 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:41403 every 8 connection(s)
I20260927 21:42:24.914570 13720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:42:24.922349 13788 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:42:24.924261 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 13720
I20260927 21:42:24.925180 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:46475
--local_ip_for_outbound_sockets=127.0.202.129
--tserver_master_addrs=127.0.202.190:41403
--webserver_port=35335
--webserver_interface=127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:24.926164 13788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Bootstrap starting.
I20260927 21:42:24.947067 13788 log.cc:824] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:24.957429 13788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:42:24.958066 13788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Bootstrap complete.
I20260927 21:42:24.968766 13788 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:24.969751 13788 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed70ae400274a798c6fe0e68ff047e4, State: Initialized, Role: FOLLOWER
I20260927 21:42:24.970453 13788 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:24.970746 13788 raft_consensus.cc:388] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:42:24.970922 13788 raft_consensus.cc:482] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:42:24.971127 13788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:42:24.975461 13788 raft_consensus.cc:504] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:24.976032 13788 leader_election.cc:302] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fed70ae400274a798c6fe0e68ff047e4; no voters:
I20260927 21:42:24.976778 13788 leader_election.cc:288] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:42:24.977066 13792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:42:24.978283 13792 raft_consensus.cc:686] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [term 2 LEADER]: Becoming Leader. State: Replica: fed70ae400274a798c6fe0e68ff047e4, State: Running, Role: LEADER
I20260927 21:42:24.979154 13788 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:24.979094 13792 consensus_queue.cc:237] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } }
I20260927 21:42:24.981916 13793 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fed70ae400274a798c6fe0e68ff047e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } } }
I20260927 21:42:24.982336 13793 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:24.987278 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fed70ae400274a798c6fe0e68ff047e4. Latest consensus state: current_term: 2 leader_uuid: "fed70ae400274a798c6fe0e68ff047e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed70ae400274a798c6fe0e68ff047e4" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41403 } } }
I20260927 21:42:24.988204 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:24.990968 13797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:42:24.998040 13797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:42:24.999966 13797 catalog_manager.cc:1259] Loaded cluster ID: 95cef6d27f5e4e63960239996347b8db
I20260927 21:42:25.000170 13797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:42:25.005507 13797 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:42:25.007575 13797 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Loaded TSK: 0
I20260927 21:42:25.009634 13797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:42:25.010515 13788 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 21:42:25.409338 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:25.409782 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:25.410020 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:25.423305 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:25.423533 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:25.423769 13790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:42:25.441380 13790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:46475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=35335
--tserver_master_addrs=127.0.202.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:25.443468 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:25.445595 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:25.481472 13826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:25.484892 13790 server_base.cc:1034] running on GCE node
W20260927 21:42:25.485361 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:25.486413 13825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:25.487504 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:25.504398 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:25.505831 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790545345505740 us; error 99 us; skew 500 ppm
I20260927 21:42:25.506286 13790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:25 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:25.515954 13790 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:42:25.535797 13790 webserver.cc:460] Webserver started at http://127.0.202.129:35335/ using document root <none> and password file <none>
I20260927 21:42:25.537142 13790 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:25.537422 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:42:25 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:25.575608 13790 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 21:42:25.585450 13837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:25.587882 13790 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.009s
I20260927 21:42:25.588127 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "9129963a46664aeeb00fa9f6f16c46f8"
format_stamp: "Formatted at 2026-09-27 21:42:22 on dist-test-slave-25dq"
server_key: "6fc136a1fe23a60909c9542d25fe6b23"
server_key_iv: "5c651d07431d5ed7d1da066950a73a15"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:25.589733 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:25.627446 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:25.637347 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:25.637746 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:25.645167 13790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:25.653374 13790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:25.653609 13790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:42:25.653795 13790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:25.653920 13790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:25.704730 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:46475
I20260927 21:42:25.705276 13949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:46475 every 8 connection(s)
I20260927 21:42:25.706390 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:42:25.707104 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 13790
I20260927 21:42:25.708103 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:40775
--local_ip_for_outbound_sockets=127.0.202.130
--tserver_master_addrs=127.0.202.190:41403
--webserver_port=40611
--webserver_interface=127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:42:25 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:25.748008 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:25.710737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41403 (local address 127.0.202.129:42197)
0927 21:42:25.713333 (+ 2596us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:25.713395 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:25.716229 (+ 2834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:25.721361 (+ 5132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:25.721373 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:25.722030 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:25.722832 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:25.722858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:25.729107 (+ 6249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:25.729115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:25.731362 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:25.731386 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:25.733191 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:25.733201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:25.733512 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:25.734424 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:25.734464 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:25.737417 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:25.742741 (+ 5324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:25.742752 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:25.742770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:25.743396 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:25.743973 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:25.743978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:25.743982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:25.744219 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:25.745255 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:25.745263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:25.745697 (+ 434us) client_negotiation.cc:769] Sending connection context
0927 21:42:25.745995 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 21:42:25.746561 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2121,"thread_start_us":1864,"threads_started":1}
I20260927 21:42:25.751116 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:25.711769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:42197 (local address 127.0.202.190:41403)
0927 21:42:25.718584 (+ 6815us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:25.718605 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:25.718668 (+ 63us) server_negotiation.cc:407] Connection header received
0927 21:42:25.719594 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:25.719614 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:25.720145 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:25.720616 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:25.723142 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:25.728822 (+ 5680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:25.731654 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:25.732874 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:25.737795 (+ 4921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:25.738209 (+ 414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:25.738220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:25.738244 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:25.738292 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:25.742528 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:25.743620 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:25.743629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:25.743639 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:25.743834 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:25.744399 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:25.744404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:25.744408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:25.745088 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:25.748977 (+ 3889us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:25.749251 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:42:25.749692 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6476,"thread_start_us":6226,"threads_started":1}
I20260927 21:42:25.755643 13950 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41403
I20260927 21:42:25.756002 13950 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:25.756994 13950 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
W20260927 21:42:25.760586 13950 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:41403 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:42:25.760840 13950 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:25.761153 13950 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
I20260927 21:42:25.762845 13950 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:25.763235 13950 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
W20260927 21:42:25.764604 13950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:42:26.166554 13954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:26.167035 13954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:26.167246 13954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:26.178586 13954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:26.178762 13954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:26.178943 13954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:42:26.189877 13954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:40775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=40611
--tserver_master_addrs=127.0.202.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:26.191393 13954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:26.192855 13954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:26.210254 13970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:26.217027 13971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:26.217545 13973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:26.218026 13954 server_base.cc:1034] running on GCE node
I20260927 21:42:26.219018 13954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:26.220336 13954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:26.221644 13954 hybrid_clock.cc:648] HybridClock initialized: now 1790545346221577 us; error 51 us; skew 500 ppm
I20260927 21:42:26.222019 13954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:26 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:26.226164 13954 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:42:26.230382 13954 webserver.cc:460] Webserver started at http://127.0.202.130:40611/ using document root <none> and password file <none>
I20260927 21:42:26.231366 13954 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:26.231570 13954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:42:26 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
Sep 27 21:42:26 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790545346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:26.294319 13954 fs_manager.cc:709] Time spent opening directory manager: real 0.040s user 0.020s sys 0.018s
I20260927 21:42:26.298290 13981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:26.299918 13954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:42:26.300119 13954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "be0eece6578b49818b9d4313d9b537cb"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "3f33117eb0faa830cc6db1878f7f4029"
server_key_iv: "6890f48309db5a9ee23a1c9f89ab351c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:26.301292 13954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:26.338342 13954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:26.343861 13954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:26.344241 13954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:26.345409 13954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:26.346854 13954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:26.346983 13954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:26.347127 13954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:26.347218 13954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:26.419988 13954 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:40775
I20260927 21:42:26.422009 13954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:42:26.420050 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:40775 every 8 connection(s)
I20260927 21:42:26.431979 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 13954
I20260927 21:42:26.432965 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.202.148:46365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:37149
--local_ip_for_outbound_sockets=127.0.202.131
--tserver_master_addrs=127.0.202.190:41403
--webserver_port=39855
--webserver_interface=127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:42:26 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:26.482337 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:26.429765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:40187 (local address 127.0.202.190:41403)
0927 21:42:26.451737 (+ 21972us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:26.451744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:26.454529 (+ 2785us) server_negotiation.cc:407] Connection header received
0927 21:42:26.461467 (+ 6938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:26.461474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:26.461588 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:26.461677 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:26.464744 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:26.465968 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:26.468769 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:26.469409 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:26.469530 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:26.474072 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:26.474080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:26.474085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:26.474131 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:26.478070 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:26.479144 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:26.479150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:26.479153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:26.479306 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:26.479909 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:26.479914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:26.479918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:26.480346 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:26.480494 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:26.481676 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0927 21:42:26.481937 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21800,"thread_start_us":21986,"threads_started":1}
I20260927 21:42:26.485064 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:26.429135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41403 (local address 127.0.202.130:40187)
0927 21:42:26.452548 (+ 23413us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:26.452614 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:26.455820 (+ 3206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:26.462284 (+ 6464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:26.462298 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:26.462993 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:26.463857 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:26.463883 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:26.466189 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:26.466195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:26.468573 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:26.468588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:26.469929 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:26.469936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:26.470224 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:26.471066 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:26.471097 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:26.473784 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:26.478268 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:26.478280 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:26.478300 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:26.478991 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:26.479461 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:26.479466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:26.479469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:26.479770 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:26.480661 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:26.480670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:26.481141 (+ 471us) client_negotiation.cc:769] Sending connection context
0927 21:42:26.482895 (+ 1754us) client_negotiation.cc:240] Negotiation successful
0927 21:42:26.483506 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22990,"thread_start_us":31142,"threads_started":1}
I20260927 21:42:26.487259 14095 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41403
I20260927 21:42:26.487632 14095 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:26.488621 14095 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
I20260927 21:42:26.490882 13753 ts_manager.cc:194] Registered new tserver with Master: be0eece6578b49818b9d4313d9b537cb (127.0.202.130:40775)
I20260927 21:42:26.492781 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:40187
Sep 27 21:42:26 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:26.764742 13950 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:26.765226 13950 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
I20260927 21:42:26.766580 13753 ts_manager.cc:194] Registered new tserver with Master: 9129963a46664aeeb00fa9f6f16c46f8 (127.0.202.129:46475)
I20260927 21:42:26.767830 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:42197
W20260927 21:42:26.967769 14099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:26.968338 14099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:26.968547 14099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:26.980270 14099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:26.980545 14099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:26.980805 14099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:42:26.998440 14099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.202.148:46365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:37149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=39855
--tserver_master_addrs=127.0.202.190:41403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:27.000512 14099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:27.002365 14099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:27.073968 14104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:27.075733 14105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:27.089941 14107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:27.090801 14099 server_base.cc:1034] running on GCE node
I20260927 21:42:27.091604 14099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:27.092965 14099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:27.094305 14099 hybrid_clock.cc:648] HybridClock initialized: now 1790545347094236 us; error 54 us; skew 500 ppm
I20260927 21:42:27.094741 14099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:27 dist-test-slave-25dq krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:27.101025 14099 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:42:27.106108 14099 webserver.cc:460] Webserver started at http://127.0.202.131:39855/ using document root <none> and password file <none>
I20260927 21:42:27.107116 14099 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:27.107348 14099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:42:27 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:27.139540 14099 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.003s
I20260927 21:42:27.146067 14116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:27.148187 14099 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260927 21:42:27.148435 14099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "36dadcbcb66d4e0496c674bc2b9a059b"
format_stamp: "Formatted at 2026-09-27 21:42:23 on dist-test-slave-25dq"
server_key: "8e428c29899fc4e56419ce9e32a9f688"
server_key_iv: "901e486df3bd070b8629721cb397779c"
server_key_version: "kuduclusterkey@0"
I20260927 21:42:27.149938 14099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 21:42:27 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for HTTP/127.0.202.148@KRBTEST.COM
I20260927 21:42:27.185719 14099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:27.192688 14099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:27.193089 14099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:27.194705 14099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:27.196722 14099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:27.196856 14099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:27.197003 14099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:27.197110 14099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:27.288164 14228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37149 every 8 connection(s)
I20260927 21:42:27.288558 14099 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37149
I20260927 21:42:27.290584 14099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:42:27.293702 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14099
I20260927 21:42:27.320839 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.295268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.0.202.190:41403)
0927 21:42:27.296711 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.296721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.297299 (+ 578us) server_negotiation.cc:407] Connection header received
0927 21:42:27.297554 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.297563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.297704 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.297833 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:27.299705 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.301311 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.302760 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.303683 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.303840 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.308035 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:27.308043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:27.308048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:27.308097 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:27.314918 (+ 6821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.317690 (+ 2772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.317695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.317699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.317872 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.318566 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.318572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.318576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.318955 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:27.319275 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.320338 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.320578 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":1149,"threads_started":1}
Sep 27 21:42:27 dist-test-slave-25dq krb5kdc[12741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:27.354511 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.308551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:45901 (local address 127.0.202.190:41403)
0927 21:42:27.316807 (+ 8256us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.316815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.317749 (+ 934us) server_negotiation.cc:407] Connection header received
0927 21:42:27.331766 (+ 14017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.331775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.331883 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.331992 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:27.334849 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.336369 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.339201 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.339902 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.340035 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.345089 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:27.345097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:27.345103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:27.345163 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:27.349004 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.350742 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.350749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.350753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.350932 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.351924 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.351930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.351934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.352407 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:27.352563 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.353743 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.354042 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8088,"thread_start_us":7922,"threads_started":1}
I20260927 21:42:27.357141 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.309351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:41403 (local address 127.0.202.131:45901)
0927 21:42:27.317517 (+ 8166us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:27.317614 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:27.331258 (+ 13644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:27.332813 (+ 1555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:27.332828 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:27.333547 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:27.334550 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.334582 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.336614 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.336619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.338998 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.339026 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.340456 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.340465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.340782 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.341840 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:27.341880 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:27.344779 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:27.349188 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:27.349199 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:27.349222 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:27.350551 (+ 1329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:27.351462 (+ 911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:27.351467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:27.351471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:27.351745 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:27.352670 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:27.352679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:27.353221 (+ 542us) client_negotiation.cc:769] Sending connection context
0927 21:42:27.354773 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0927 21:42:27.355452 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7782,"thread_start_us":7524,"threads_started":1}
I20260927 21:42:27.359002 14229 heartbeater.cc:347] Connected to a master server at 127.0.202.190:41403
I20260927 21:42:27.359366 14229 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:27.360394 14229 heartbeater.cc:511] Master 127.0.202.190:41403 requested a full tablet report, sending...
I20260927 21:42:27.362802 13752 ts_manager.cc:194] Registered new tserver with Master: 36dadcbcb66d4e0496c674bc2b9a059b (127.0.202.131:37149)
I20260927 21:42:27.363983 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:42:27.364188 13752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:45901
I20260927 21:42:27.397187 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.380246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.0.202.190:41403)
0927 21:42:27.380477 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.380483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.381027 (+ 544us) server_negotiation.cc:407] Connection header received
0927 21:42:27.381320 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.381327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.381424 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.381506 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:27.383963 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.385239 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.386710 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.387436 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.387584 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.390372 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:27.390379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:27.390385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:27.390433 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:27.394094 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.395024 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.395039 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.395043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.395184 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.395566 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.395572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.395575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.395951 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:27.396206 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.396604 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.396831 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 21:42:27.403448 13752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:42:27.415421 13752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:42:27.458590 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.434930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.0.202.129:46475)
0927 21:42:27.448837 (+ 13907us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.448846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.448876 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:42:27.448993 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.449004 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.449346 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.449598 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:27.451505 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.453780 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.456162 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.457310 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.457485 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.457746 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.458049 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.458225 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13730,"thread_start_us":13560,"threads_started":1}
I20260927 21:42:27.458679 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.434281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:46475 (local address 127.0.0.1:48104)
0927 21:42:27.447679 (+ 13398us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:27.447736 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:27.447932 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:27.449962 (+ 2030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:27.449974 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:27.450048 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:27.451184 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.451205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.453950 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.453956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.455948 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.455961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.457607 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.457613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.457812 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.457852 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:42:27.457995 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 21:42:27.458274 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13098,"mutex_wait_us":6,"spinlock_wait_cycles":1792,"thread_start_us":14075,"threads_started":1}
I20260927 21:42:27.460557 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.435153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.0.202.131:37149)
0927 21:42:27.443389 (+ 8236us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.443399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.448916 (+ 5517us) server_negotiation.cc:407] Connection header received
0927 21:42:27.449073 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.449081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.449334 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.449563 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:27.451784 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.453495 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.456559 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.458286 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.458400 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.458641 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.459937 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.460128 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8048,"thread_start_us":7888,"threads_started":1}
I20260927 21:42:27.460635 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.448858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:40775 (local address 127.0.0.1:47818)
0927 21:42:27.451326 (+ 2468us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:27.451344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:27.451512 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:27.452493 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:27.452498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:27.452524 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:27.452922 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.452943 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.454792 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.454797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.457495 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.457511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.459795 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.459804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.459984 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.460016 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:42:27.460112 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:42:27.460254 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2341,"thread_start_us":2175,"threads_started":1}
I20260927 21:42:27.461418 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.434464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:37149 (local address 127.0.0.1:44234)
0927 21:42:27.448654 (+ 14190us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:27.448672 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:27.448788 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:27.449978 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:27.449986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:27.450219 (+ 233us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:27.451484 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.451498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.453655 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.453661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.456368 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.456385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.459472 (+ 3087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.459478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.459688 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.459737 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:42:27.460859 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0927 21:42:27.461012 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14055,"mutex_wait_us":171,"spinlock_wait_cycles":13568,"thread_start_us":12776,"threads_started":1}
I20260927 21:42:27.461586 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.449323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.0.202.130:40775)
0927 21:42:27.450906 (+ 1583us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.450919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.451468 (+ 549us) server_negotiation.cc:407] Connection header received
0927 21:42:27.451678 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.451688 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.452016 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.452232 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:27.453122 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.454638 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.457712 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.459242 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.459390 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.459631 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.461004 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.461202 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1379,"thread_start_us":1197,"threads_started":1}
I20260927 21:42:27.462985 13885 tablet_service.cc:1467] Processing CreateTablet for tablet f3a021c660da4d439d29f89c486f0974 (DEFAULT_TABLE table=test-table [id=07ac243eb63a44c4be57ff0124a109a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:27.464649 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a021c660da4d439d29f89c486f0974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:27.465543 14030 tablet_service.cc:1467] Processing CreateTablet for tablet f3a021c660da4d439d29f89c486f0974 (DEFAULT_TABLE table=test-table [id=07ac243eb63a44c4be57ff0124a109a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:27.467154 14030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a021c660da4d439d29f89c486f0974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:27.467445 14164 tablet_service.cc:1467] Processing CreateTablet for tablet f3a021c660da4d439d29f89c486f0974 (DEFAULT_TABLE table=test-table [id=07ac243eb63a44c4be57ff0124a109a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:27.468864 14164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a021c660da4d439d29f89c486f0974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:27.481256 14257 tablet_bootstrap.cc:492] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Bootstrap starting.
I20260927 21:42:27.483505 14259 tablet_bootstrap.cc:492] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Bootstrap starting.
I20260927 21:42:27.484332 14258 tablet_bootstrap.cc:492] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Bootstrap starting.
I20260927 21:42:27.485453 14257 tablet_bootstrap.cc:654] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:27.486909 14257 log.cc:824] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:27.488281 14258 tablet_bootstrap.cc:654] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:27.489751 14258 log.cc:824] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:27.489854 14257 tablet_bootstrap.cc:492] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: No bootstrap required, opened a new log
I20260927 21:42:27.490167 14257 ts_tablet_manager.cc:1397] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20260927 21:42:27.492069 14259 tablet_bootstrap.cc:654] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:27.493880 14259 log.cc:824] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:27.497320 14259 tablet_bootstrap.cc:492] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: No bootstrap required, opened a new log
I20260927 21:42:27.497627 14259 ts_tablet_manager.cc:1397] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260927 21:42:27.497706 14258 tablet_bootstrap.cc:492] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: No bootstrap required, opened a new log
I20260927 21:42:27.497970 14258 ts_tablet_manager.cc:1397] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 21:42:27.498091 14095 heartbeater.cc:503] Master 127.0.202.190:41403 was elected leader, sending a full tablet report...
I20260927 21:42:27.498986 14257 raft_consensus.cc:348] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.500996 14257 raft_consensus.cc:374] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:27.501132 14257 raft_consensus.cc:729] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36dadcbcb66d4e0496c674bc2b9a059b, State: Initialized, Role: FOLLOWER
I20260927 21:42:27.501811 14257 consensus_queue.cc:260] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.504408 14229 heartbeater.cc:503] Master 127.0.202.190:41403 was elected leader, sending a full tablet report...
I20260927 21:42:27.505179 14257 ts_tablet_manager.cc:1428] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260927 21:42:27.508639 14258 raft_consensus.cc:348] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.509294 14258 raft_consensus.cc:374] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:27.509408 14258 raft_consensus.cc:729] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9129963a46664aeeb00fa9f6f16c46f8, State: Initialized, Role: FOLLOWER
I20260927 21:42:27.509343 14259 raft_consensus.cc:348] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.509924 14259 raft_consensus.cc:374] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:27.510079 14259 raft_consensus.cc:729] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0eece6578b49818b9d4313d9b537cb, State: Initialized, Role: FOLLOWER
I20260927 21:42:27.510124 14258 consensus_queue.cc:260] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.510807 14259 consensus_queue.cc:260] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.511384 13950 heartbeater.cc:503] Master 127.0.202.190:41403 was elected leader, sending a full tablet report...
I20260927 21:42:27.512012 14258 ts_tablet_manager.cc:1428] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 21:42:27.512619 14259 ts_tablet_manager.cc:1428] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260927 21:42:27.575045 14234 tablet.cc:2369] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:42:27.618772 14264 raft_consensus.cc:482] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:42:27.619093 14264 raft_consensus.cc:504] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.621068 14264 leader_election.cc:288] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36dadcbcb66d4e0496c674bc2b9a059b (127.0.202.131:37149), be0eece6578b49818b9d4313d9b537cb (127.0.202.130:40775)
I20260927 21:42:27.630275 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.621645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:51067 (local address 127.0.202.130:40775)
0927 21:42:27.621871 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.621878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.622737 (+ 859us) server_negotiation.cc:407] Connection header received
0927 21:42:27.622931 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.622936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.623006 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.623077 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:27.623765 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.624849 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.628074 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.629244 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.629373 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.629455 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.629855 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.629995 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:42:27.630362 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.621592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:58585 (local address 127.0.202.131:37149)
0927 21:42:27.622123 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.622129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.622348 (+ 219us) server_negotiation.cc:407] Connection header received
0927 21:42:27.622593 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.622598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.622663 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.622774 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:27.623553 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.624738 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.628099 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.629360 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.629545 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.629621 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.629962 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.630115 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260927 21:42:27.630470 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.621466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:37149 (local address 127.0.202.129:58585)
0927 21:42:27.622241 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:27.622257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:27.622450 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:27.622819 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:27.622827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:27.622859 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:27.623361 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.623380 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.624882 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.624887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.627860 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.627877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.629610 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.629616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.629755 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.629812 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:42:27.629909 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:42:27.630032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":193,"thread_start_us":353,"threads_started":1}
I20260927 21:42:27.630493 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.621599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:40775 (local address 127.0.202.129:51067)
0927 21:42:27.622612 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:27.622632 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:27.622787 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:27.623140 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:27.623147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:27.623181 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:27.623609 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.623618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.625027 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.625035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.627860 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:27.627877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.629466 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.629473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:27.629686 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.629716 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:42:27.629799 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:42:27.629959 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":54,"thread_start_us":684,"threads_started":1}
I20260927 21:42:27.631662 14184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a021c660da4d439d29f89c486f0974" candidate_uuid: "9129963a46664aeeb00fa9f6f16c46f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" is_pre_election: true
I20260927 21:42:27.631649 14050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a021c660da4d439d29f89c486f0974" candidate_uuid: "9129963a46664aeeb00fa9f6f16c46f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0eece6578b49818b9d4313d9b537cb" is_pre_election: true
I20260927 21:42:27.632174 14050 raft_consensus.cc:2454] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9129963a46664aeeb00fa9f6f16c46f8 in term 0.
I20260927 21:42:27.632205 14184 raft_consensus.cc:2454] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9129963a46664aeeb00fa9f6f16c46f8 in term 0.
I20260927 21:42:27.633020 13838 leader_election.cc:302] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36dadcbcb66d4e0496c674bc2b9a059b, 9129963a46664aeeb00fa9f6f16c46f8, be0eece6578b49818b9d4313d9b537cb; no voters:
I20260927 21:42:27.633543 14264 raft_consensus.cc:2781] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:42:27.633733 14264 raft_consensus.cc:482] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:42:27.633883 14264 raft_consensus.cc:3037] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:27.636601 14264 raft_consensus.cc:504] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.637442 14264 leader_election.cc:288] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [CANDIDATE]: Term 1 election: Requested vote from peers 36dadcbcb66d4e0496c674bc2b9a059b (127.0.202.131:37149), be0eece6578b49818b9d4313d9b537cb (127.0.202.130:40775)
I20260927 21:42:27.638046 14184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a021c660da4d439d29f89c486f0974" candidate_uuid: "9129963a46664aeeb00fa9f6f16c46f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36dadcbcb66d4e0496c674bc2b9a059b"
I20260927 21:42:27.638136 14050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a021c660da4d439d29f89c486f0974" candidate_uuid: "9129963a46664aeeb00fa9f6f16c46f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0eece6578b49818b9d4313d9b537cb"
I20260927 21:42:27.638291 14184 raft_consensus.cc:3037] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:27.638387 14050 raft_consensus.cc:3037] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:27.640920 14184 raft_consensus.cc:2454] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9129963a46664aeeb00fa9f6f16c46f8 in term 1.
I20260927 21:42:27.641036 14050 raft_consensus.cc:2454] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9129963a46664aeeb00fa9f6f16c46f8 in term 1.
I20260927 21:42:27.641505 13840 leader_election.cc:302] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36dadcbcb66d4e0496c674bc2b9a059b, 9129963a46664aeeb00fa9f6f16c46f8; no voters:
I20260927 21:42:27.641907 14264 raft_consensus.cc:2781] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:42:27.642544 14264 raft_consensus.cc:686] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9129963a46664aeeb00fa9f6f16c46f8, State: Running, Role: LEADER
I20260927 21:42:27.643067 14264 consensus_queue.cc:237] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } }
I20260927 21:42:27.647990 13753 catalog_manager.cc:5495] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9129963a46664aeeb00fa9f6f16c46f8 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "9129963a46664aeeb00fa9f6f16c46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9129963a46664aeeb00fa9f6f16c46f8" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 46475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 } health_report { overall_health: UNKNOWN } } }
I20260927 21:42:27.688669 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.682535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.0.202.129:46475)
0927 21:42:27.682736 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.682741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.682758 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:42:27.682868 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.682872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.683024 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.683168 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:42:27.683563 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.684543 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.686302 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.686765 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.686890 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.687460 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:42:27.687994 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:42:27.688120 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.688364 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.688441 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:42:27.697504 14050 raft_consensus.cc:1261] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 1 FOLLOWER]: Refusing update from remote peer 9129963a46664aeeb00fa9f6f16c46f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:27.697500 14184 raft_consensus.cc:1261] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 1 FOLLOWER]: Refusing update from remote peer 9129963a46664aeeb00fa9f6f16c46f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:27.698561 14268 consensus_queue.cc:1035] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0eece6578b49818b9d4313d9b537cb" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 40775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:27.698872 14264 consensus_queue.cc:1035] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dadcbcb66d4e0496c674bc2b9a059b" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:27.712428 14271 mvcc.cc:205] Tried to move back new op lower bound from 7334073744157736960 to 7334073743950381056. Current Snapshot: MvccSnapshot[applied={T|T < 7334073744157736960}]
W20260927 21:42:27.716120 14096 tablet.cc:2369] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:42:27.723609 14273 mvcc.cc:205] Tried to move back new op lower bound from 7334073744157736960 to 7334073743950381056. Current Snapshot: MvccSnapshot[applied={T|T < 7334073744157736960}]
I20260927 21:42:27.729640 13753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:42:27.729959 13753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:42:27.733999 13753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fed70ae400274a798c6fe0e68ff047e4: Sending DeleteTablet for 3 replicas of tablet f3a021c660da4d439d29f89c486f0974
I20260927 21:42:27.735245 14164 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a021c660da4d439d29f89c486f0974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:27 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:44234
I20260927 21:42:27.735289 14030 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a021c660da4d439d29f89c486f0974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:27 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:47818
I20260927 21:42:27.735667 13885 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a021c660da4d439d29f89c486f0974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:27 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:48104
I20260927 21:42:27.736400 14281 tablet_replica.cc:330] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: stopping tablet replica
I20260927 21:42:27.736819 14281 raft_consensus.cc:2229] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:42:27.737372 14281 raft_consensus.cc:2258] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:27.738281 14283 tablet_replica.cc:330] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: stopping tablet replica
I20260927 21:42:27.738679 14283 raft_consensus.cc:2229] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:42:27.739324 14281 ts_tablet_manager.cc:1905] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:27.739317 14283 raft_consensus.cc:2258] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:27.741467 14283 ts_tablet_manager.cc:1905] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:27.742645 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 13790
I20260927 21:42:27.747948 14281 ts_tablet_manager.cc:1918] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:27.748153 14281 log.cc:1196] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/f3a021c660da4d439d29f89c486f0974
I20260927 21:42:27.748828 14281 ts_tablet_manager.cc:1939] T f3a021c660da4d439d29f89c486f0974 P be0eece6578b49818b9d4313d9b537cb: Deleting consensus metadata
I20260927 21:42:27.748886 14283 ts_tablet_manager.cc:1918] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:27.749100 14283 log.cc:1196] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/f3a021c660da4d439d29f89c486f0974
I20260927 21:42:27.749625 14283 ts_tablet_manager.cc:1939] T f3a021c660da4d439d29f89c486f0974 P 9129963a46664aeeb00fa9f6f16c46f8: Deleting consensus metadata
I20260927 21:42:27.751185 13737 catalog_manager.cc:4841] TS be0eece6578b49818b9d4313d9b537cb (127.0.202.130:40775): tablet f3a021c660da4d439d29f89c486f0974 (table test-table [id=07ac243eb63a44c4be57ff0124a109a5]) successfully deleted
I20260927 21:42:27.751886 13737 catalog_manager.cc:4841] TS 9129963a46664aeeb00fa9f6f16c46f8 (127.0.202.129:46475): tablet f3a021c660da4d439d29f89c486f0974 (table test-table [id=07ac243eb63a44c4be57ff0124a109a5]) successfully deleted
I20260927 21:42:27.755712 14282 tablet_replica.cc:330] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: stopping tablet replica
I20260927 21:42:27.756458 14282 raft_consensus.cc:2229] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:42:27.757032 14282 raft_consensus.cc:2258] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:27.759336 14282 ts_tablet_manager.cc:1905] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:27.763355 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.744424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.0.202.129:46475)
0927 21:42:27.744592 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.744598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.744616 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:42:27.744671 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.744676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.744751 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.744838 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:27.747531 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.748772 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.751744 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.752482 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.752626 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.756618 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:27.756637 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:27.756658 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:27.756703 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:27.760244 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.761175 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.761184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.761194 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.761320 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.761713 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.761719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.761722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.762115 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:27.762402 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.762722 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.762955 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:42:27.764038 13945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:27.766520 14282 ts_tablet_manager.cc:1918] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:27.766862 14282 log.cc:1196] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/f3a021c660da4d439d29f89c486f0974
I20260927 21:42:27.767633 14282 ts_tablet_manager.cc:1939] T f3a021c660da4d439d29f89c486f0974 P 36dadcbcb66d4e0496c674bc2b9a059b: Deleting consensus metadata
I20260927 21:42:27.769997 13739 catalog_manager.cc:4841] TS 36dadcbcb66d4e0496c674bc2b9a059b (127.0.202.131:37149): tablet f3a021c660da4d439d29f89c486f0974 (table test-table [id=07ac243eb63a44c4be57ff0124a109a5]) successfully deleted
I20260927 21:42:27.862578 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 13790
I20260927 21:42:27.881496 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 13954
I20260927 21:42:27.896030 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:27.882497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.0.202.130:40775)
0927 21:42:27.882724 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:27.882730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:27.882751 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:42:27.882872 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:27.882879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:27.882949 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:27.883053 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:27.884481 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.885233 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.886554 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:27.886958 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:27.887107 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:27.889830 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:27.889853 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:27.889869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:27.889909 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:27.893189 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.893916 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.893924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.893926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.894067 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:27.894448 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:27.894455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:27.894458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:27.894816 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:27.895155 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:27.895439 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 21:42:27.895666 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:42:27.896857 14090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:27.988359 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 13954
I20260927 21:42:28.005120 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14099
I20260927 21:42:28.020807 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:28.005997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.0.202.131:37149)
0927 21:42:28.006267 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:28.006276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:28.006304 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:42:28.006540 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:28.006547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:28.006640 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:28.006760 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:28.008197 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:28.009051 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:28.010476 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:28.011055 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:28.011191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:28.014393 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:28.014408 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:28.014425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:28.014461 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:28.017692 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:28.018426 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:28.018435 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:28.018437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:28.018575 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:28.018979 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:28.018983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:28.018985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:28.019413 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:28.019711 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:28.020097 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 21:42:28.020279 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 21:42:28.021600 14224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:28.114584 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14099
I20260927 21:42:28.131093 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 13720
I20260927 21:42:28.132414 13783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:28.246490 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 13720
I20260927 21:42:28.268460 810 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:42:28.614578 810 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:42:28.614758 810 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:42:28.961128 810 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:42:28.961 UTC [12771] LOG: received smart shutdown request
2026-09-27 21:42:28.965 UTC [12771] LOG: background worker "logical replication launcher" (PID 12780) exited with exit code 1
2026-09-27 21:42:28.965 UTC [12775] LOG: shutting down
2026-09-27 21:42:29.178 UTC [12771] LOG: database system is shut down
2026-09-27T21:42:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56060 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:42:29 dist-test-slave-25dq krb5kdc[14296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:42:29 dist-test-slave-25dq krb5kdc[14296](info): set up 2 sockets
Sep 27 21:42:29 dist-test-slave-25dq krb5kdc[14296](info): commencing operation
krb5kdc: starting...
W20260927 21:42:30.441074 810 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:42:30 dist-test-slave-25dq krb5kdc[14296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:42:30.693205 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33921
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33921,127.0.202.189:41489,127.0.202.188:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:31.039669 14309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:31.039988 14309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:31.040386 14309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:31.048902 14309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:31.049026 14309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:31.049108 14309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:31.049194 14309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:31.049263 14309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:31.059751 14309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33921,127.0.202.189:41489,127.0.202.188:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:31.061340 14309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:31.062798 14309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:31.075313 14314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:31.075423 14315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:31.075620 14317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:31.076840 14309 server_base.cc:1034] running on GCE node
I20260927 21:42:31.077682 14309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:31.079170 14309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:31.080399 14309 hybrid_clock.cc:648] HybridClock initialized: now 1790545351080333 us; error 57 us; skew 500 ppm
I20260927 21:42:31.080767 14309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:31 dist-test-slave-25dq krb5kdc[14296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:31.091679 14309 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:42:31.093444 14309 webserver.cc:460] Webserver started at http://127.0.202.190:42961/ using document root <none> and password file <none>
I20260927 21:42:31.094213 14309 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:31.094363 14309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:31.094696 14309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:31.097936 14309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "d23aacaa95474edaad33311a22e0fcf0"
format_stamp: "Formatted at 2026-09-27 21:42:31 on dist-test-slave-25dq"
server_key: "14e414c11d11afbbde7927e44f04b613"
server_key_iv: "196a471d306f33b58d3360464d5069f3"
server_key_version: "encryptionkey@0"
I20260927 21:42:31.098773 14309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "d23aacaa95474edaad33311a22e0fcf0"
format_stamp: "Formatted at 2026-09-27 21:42:31 on dist-test-slave-25dq"
server_key: "14e414c11d11afbbde7927e44f04b613"
server_key_iv: "196a471d306f33b58d3360464d5069f3"
server_key_version: "encryptionkey@0"
I20260927 21:42:31.104281 14309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:42:31.108381 14324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:31.109894 14309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:42:31.110157 14309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "d23aacaa95474edaad33311a22e0fcf0"
format_stamp: "Formatted at 2026-09-27 21:42:31 on dist-test-slave-25dq"
server_key: "14e414c11d11afbbde7927e44f04b613"
server_key_iv: "196a471d306f33b58d3360464d5069f3"
server_key_version: "encryptionkey@0"
I20260927 21:42:31.110396 14309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:31.130144 14309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:31.134996 14309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:31.135375 14309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:31.156010 14309 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:33921
I20260927 21:42:31.156032 14375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:33921 every 8 connection(s)
I20260927 21:42:31.157546 14309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:42:31.158768 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14309
I20260927 21:42:31.159026 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:42:31.159795 810 external_mini_cluster.cc:1428] Setting key 3ece3eeb373b8591f4530dce652e9c39
I20260927 21:42:31.162508 14376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:31.171152 14376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.202.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.189@KRBTEST.COM" created.
Sep 27 21:42:31 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:31.197641 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.174491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.202.190:33921)
0927 21:42:31.175442 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:31.175459 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:31.176003 (+ 544us) server_negotiation.cc:407] Connection header received
0927 21:42:31.178190 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:31.178200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:31.178542 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:31.178808 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:31.180092 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.182135 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.183984 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.184794 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.184958 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.189095 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:31.189110 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:31.189130 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:31.189180 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:31.193086 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.194080 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.194087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.194090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.194267 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.194755 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.194758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.194760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.195302 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:31.195456 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:31.196159 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 21:42:31.196542 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":651,"thread_start_us":410,"threads_started":1}
I20260927 21:42:31.197669 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.174491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33921 (local address 127.0.0.1:48342)
0927 21:42:31.175820 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.175863 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:31.177688 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:31.178865 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:31.178870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:31.179494 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:31.179863 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.179885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.182325 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.182330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.183680 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.183693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.185207 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.185220 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.185479 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.186276 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:31.186301 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:31.188813 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:31.193250 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.193269 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.193286 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.193920 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.194400 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.194408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.194415 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.194633 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.195463 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:31.195473 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:31.195745 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 21:42:31.196062 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 21:42:31.196749 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":146,"spinlock_wait_cycles":45440,"thread_start_us":822,"threads_started":1}
I20260927 21:42:31.200749 14376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:42:31.202001 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.201590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44658)
0927 21:42:31.201782 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.201889 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 21:42:31.202405 14328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
I20260927 21:42:31.204020 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.203659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44662)
0927 21:42:31.203822 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.203888 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 21:42:31.204260 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:41489: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab.
Entry for principal kudu/127.0.202.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab.
I20260927 21:42:31.219919 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:41489
--webserver_interface=127.0.202.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab
--principal=kudu/127.0.202.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33921,127.0.202.189:41489,127.0.202.188:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:31.253706 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } attempt: 1
I20260927 21:42:31.255071 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.254551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44672)
0927 21:42:31.254772 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.254893 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 21:42:31.256176 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.255814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44676)
0927 21:42:31.255966 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.256043 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:42:31.256438 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:41489: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
I20260927 21:42:31.324741 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } attempt: 2
I20260927 21:42:31.326160 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.325604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44686)
0927 21:42:31.325887 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.326005 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:42:31.327493 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.326967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44702)
0927 21:42:31.327281 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.327357 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260927 21:42:31.327728 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:41489: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
I20260927 21:42:31.419003 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } attempt: 3
I20260927 21:42:31.420161 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.419724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44708)
0927 21:42:31.419931 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.420031 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:42:31.421370 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.421008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44720)
0927 21:42:31.421165 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.421248 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 21:42:31.421614 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:41489: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
W20260927 21:42:31.563495 14383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:31.563827 14383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:31.564004 14383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:42:31.564870 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } attempt: 4
I20260927 21:42:31.566262 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.565649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44734)
0927 21:42:31.565959 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.566066 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260927 21:42:31.567597 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.567090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44744)
0927 21:42:31.567354 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.567455 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260927 21:42:31.567974 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:41489: Network error: Client connection negotiation failed: client connection to 127.0.202.189:41489: connect: Connection refused (error 111)
W20260927 21:42:31.572674 14383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:31.572813 14383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:31.572908 14383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:31.573006 14383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:31.573087 14383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:31.583899 14383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33921,127.0.202.189:41489,127.0.202.188:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab
--principal=kudu/127.0.202.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.202.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:31.585402 14383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:31.586781 14383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:31.599413 14388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:31.599416 14391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:31.599447 14389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:31.600862 14383 server_base.cc:1034] running on GCE node
I20260927 21:42:31.601645 14383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:31.603219 14383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:31.604441 14383 hybrid_clock.cc:648] HybridClock initialized: now 1790545351604388 us; error 66 us; skew 500 ppm
I20260927 21:42:31.604815 14383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:31 dist-test-slave-25dq krb5kdc[14296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:31.608270 14383 init.cc:377] Logged in from keytab as kudu/127.0.202.189@KRBTEST.COM (short username kudu)
I20260927 21:42:31.609788 14383 webserver.cc:460] Webserver started at http://127.0.202.189:34321/ using document root <none> and password file <none>
I20260927 21:42:31.610556 14383 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:31.610703 14383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:31.611032 14383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:31.614213 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data/instance:
uuid: "6a6cff6704c441e9aa161190b6515e32"
format_stamp: "Formatted at 2026-09-27 21:42:31 on dist-test-slave-25dq"
server_key: "cc434e1e5b1f5006d6e6b9778762f00c"
server_key_iv: "20b5525aa21fbe65f04d437bd428e056"
server_key_version: "encryptionkey@0"
I20260927 21:42:31.615108 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/wal/instance:
uuid: "6a6cff6704c441e9aa161190b6515e32"
format_stamp: "Formatted at 2026-09-27 21:42:31 on dist-test-slave-25dq"
server_key: "cc434e1e5b1f5006d6e6b9778762f00c"
server_key_iv: "20b5525aa21fbe65f04d437bd428e056"
server_key_version: "encryptionkey@0"
I20260927 21:42:31.620755 14383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 21:42:31.624670 14398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:31.626354 14383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:42:31.626569 14383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/wal
uuid: "6a6cff6704c441e9aa161190b6515e32"
format_stamp: "Formatted at 2026-09-27 21:42:31 on dist-test-slave-25dq"
server_key: "cc434e1e5b1f5006d6e6b9778762f00c"
server_key_iv: "20b5525aa21fbe65f04d437bd428e056"
server_key_version: "encryptionkey@0"
I20260927 21:42:31.626796 14383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:31.648198 14383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:31.653493 14383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:31.653892 14383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:31.674090 14383 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.189:41489
I20260927 21:42:31.674147 14449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.189:41489 every 8 connection(s)
I20260927 21:42:31.675526 14383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
I20260927 21:42:31.679741 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:31.683990 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14383
I20260927 21:42:31.684228 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-1/wal/instance
I20260927 21:42:31.684880 810 external_mini_cluster.cc:1428] Setting key e669643471357a2cfccc935dad48da26
I20260927 21:42:31.689308 14450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:42:31 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.188@KRBTEST.COM" created.
I20260927 21:42:31.717451 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.693349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.0.202.190:33921)
0927 21:42:31.693578 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:31.693586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:31.695030 (+ 1444us) server_negotiation.cc:407] Connection header received
0927 21:42:31.697494 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:31.697499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:31.697585 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:31.697703 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:31.699889 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.700772 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.703786 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.704354 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.704499 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.709966 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:31.709976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:31.709984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:31.710038 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:31.713303 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.714277 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.714283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.714286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.714595 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.715149 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.715153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.715156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.715559 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:31.715761 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:31.716942 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0927 21:42:31.717191 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:42:31.719045 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.693855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33921 (local address 127.0.0.1:48350)
0927 21:42:31.694851 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.694900 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:31.697162 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:31.697984 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:31.697998 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:31.698821 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:31.699645 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.699685 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.700952 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.700961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.703571 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.703592 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.704637 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.704648 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.705016 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.705927 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:31.705967 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:31.709576 (+ 3609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:31.713488 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.713499 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.713519 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.714016 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.714754 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.714758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.714760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.714980 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.715763 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:31.715770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:31.716339 (+ 569us) client_negotiation.cc:769] Sending connection context
0927 21:42:31.716799 (+ 460us) client_negotiation.cc:240] Negotiation successful
0927 21:42:31.717433 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":486,"threads_started":1}
I20260927 21:42:31.721271 14450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:42:31 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:42:31.742693 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.722242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44756)
0927 21:42:31.722625 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.722650 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:31.722797 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:31.723943 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:31.723948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:31.724624 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:31.724866 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.724881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.727138 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.727143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.728034 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.728047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.729527 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.729532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.729752 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.730491 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:31.730529 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:31.733037 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:31.738216 (+ 5179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.738222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.738228 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.738934 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.739892 (+ 958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.739899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.739902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.740132 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.741394 (+ 1262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:31.741399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:31.741560 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 21:42:31.741740 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 21:42:31.742231 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":40}
I20260927 21:42:31.742805 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.722437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.0.202.189:41489)
0927 21:42:31.723436 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:31.723442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:31.723469 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:42:31.723518 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:31.723524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:31.723779 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:31.723957 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:31.725152 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.726932 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.728208 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.729333 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.729505 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.733306 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:31.733325 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:31.733349 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:31.733405 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:31.738024 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.739155 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.739166 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.739500 (+ 334us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.739708 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.740323 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.740330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.740336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.741207 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:31.741494 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:31.741895 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 21:42:31.742372 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":823,"thread_start_us":661,"threads_started":1}
I20260927 21:42:31.746546 14450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:42:31.748191 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.747382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54714)
0927 21:42:31.747815 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.747956 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 21:42:31.748694 14399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
I20260927 21:42:31.751097 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.750548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54724)
0927 21:42:31.750839 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.750943 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260927 21:42:31.751493 14450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab.
Entry for principal kudu/127.0.202.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab.
I20260927 21:42:31.757053 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:41835
--webserver_interface=127.0.202.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab
--principal=kudu/127.0.202.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33921,127.0.202.189:41489,127.0.202.188:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:31.800930 14450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 1
I20260927 21:42:31.802342 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.801780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54730)
0927 21:42:31.802011 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.802120 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 21:42:31.803404 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.803039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54734)
0927 21:42:31.803226 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.803296 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:42:31.803617 14450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
I20260927 21:42:31.867300 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } attempt: 5
I20260927 21:42:31.871852 14450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 2
I20260927 21:42:31.873040 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.872474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54736)
0927 21:42:31.872688 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.872807 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 21:42:31.874327 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.873858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54746)
0927 21:42:31.874059 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.874142 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 21:42:31.874734 14450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Sep 27 21:42:31 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:42:31.884572 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.868136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44760)
0927 21:42:31.868407 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.868429 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:31.868591 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:31.869073 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:31.869078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:31.869441 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:31.869651 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.869659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.871176 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.871181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.872566 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:31.872583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.873721 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.873729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:31.873954 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.874704 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:31.874732 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:31.877406 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:31.881865 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.881869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.881872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.882365 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.882848 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:31.882852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:31.882854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:31.882987 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:31.883718 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:31.883721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:31.883863 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 21:42:31.883987 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 21:42:31.884217 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 21:42:31.885051 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.868436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.0.202.189:41489)
0927 21:42:31.868667 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:31.868675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:31.868700 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:42:31.868803 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:31.868809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:31.868898 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:31.869027 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:31.869826 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.870981 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.872754 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:31.873536 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:31.873682 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:31.877644 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:31.877652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:31.877657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:31.877702 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:31.881671 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.882536 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.882543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.882546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.882712 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:31.883123 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:31.883129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:31.883132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:31.883554 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:31.883732 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:31.884166 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 21:42:31.884460 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:42:31.886183 14376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:42:31.887277 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.886885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54756)
0927 21:42:31.887054 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.887138 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 21:42:31.888247 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.887918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54772)
0927 21:42:31.888080 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.888148 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:42:31.888487 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
I20260927 21:42:31.939738 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 1
I20260927 21:42:31.941118 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.940574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54786)
0927 21:42:31.940781 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.940879 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 21:42:31.942415 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.942017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54792)
0927 21:42:31.942210 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.942304 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 21:42:31.942808 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
I20260927 21:42:31.966006 14450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 3
I20260927 21:42:31.967208 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.966744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54800)
0927 21:42:31.966946 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.967054 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 21:42:31.968281 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:31.967874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54812)
0927 21:42:31.968102 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:31.968177 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260927 21:42:31.968531 14450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
I20260927 21:42:32.011108 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 2
I20260927 21:42:32.012545 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.011924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54822)
0927 21:42:32.012207 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.012368 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 21:42:32.013603 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.013247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54836)
0927 21:42:32.013401 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.013482 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:42:32.013825 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
I20260927 21:42:32.111749 14450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 4
I20260927 21:42:32.113135 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.112538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54844)
0927 21:42:32.112808 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.112946 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 21:42:32.114197 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.113834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54854)
0927 21:42:32.113983 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.114089 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:42:32.114457 14450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
W20260927 21:42:32.115479 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:32.115813 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:32.115964 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:42:32.120078 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 3
I20260927 21:42:32.121213 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.120718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54870)
0927 21:42:32.120952 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.121040 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 21:42:32.122277 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.121905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54880)
0927 21:42:32.122099 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.122175 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 21:42:32.122496 14376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:41835: Network error: Client connection negotiation failed: client connection to 127.0.202.188:41835: connect: Connection refused (error 111)
W20260927 21:42:32.124831 14457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:32.124967 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:32.125062 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:32.125166 14457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:32.125240 14457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:32.135880 14457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33921,127.0.202.189:41489,127.0.202.188:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab
--principal=kudu/127.0.202.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.202.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:32.137459 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:32.138859 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:32.151087 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:32.151005 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:32.151037 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:32.152369 14457 server_base.cc:1034] running on GCE node
I20260927 21:42:32.153229 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:32.154871 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:32.156098 14457 hybrid_clock.cc:648] HybridClock initialized: now 1790545352156040 us; error 56 us; skew 500 ppm
I20260927 21:42:32.156500 14457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:32.159732 14457 init.cc:377] Logged in from keytab as kudu/127.0.202.188@KRBTEST.COM (short username kudu)
I20260927 21:42:32.161396 14457 webserver.cc:460] Webserver started at http://127.0.202.188:33897/ using document root <none> and password file <none>
I20260927 21:42:32.162065 14457 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:32.162189 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:32.162487 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:32.165531 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data/instance:
uuid: "b838d3810c9f4a16b2719ee6cab25fdc"
format_stamp: "Formatted at 2026-09-27 21:42:32 on dist-test-slave-25dq"
server_key: "56624bfac86dfc6323030984737e6988"
server_key_iv: "de25ab714a6a4ab1862472ab1b51a0fa"
server_key_version: "encryptionkey@0"
I20260927 21:42:32.166348 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/wal/instance:
uuid: "b838d3810c9f4a16b2719ee6cab25fdc"
format_stamp: "Formatted at 2026-09-27 21:42:32 on dist-test-slave-25dq"
server_key: "56624bfac86dfc6323030984737e6988"
server_key_iv: "de25ab714a6a4ab1862472ab1b51a0fa"
server_key_version: "encryptionkey@0"
I20260927 21:42:32.171754 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:42:32.175304 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:32.176801 14457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:42:32.176997 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/wal
uuid: "b838d3810c9f4a16b2719ee6cab25fdc"
format_stamp: "Formatted at 2026-09-27 21:42:32 on dist-test-slave-25dq"
server_key: "56624bfac86dfc6323030984737e6988"
server_key_iv: "de25ab714a6a4ab1862472ab1b51a0fa"
server_key_version: "encryptionkey@0"
I20260927 21:42:32.177289 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:32.190665 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:32.195729 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:32.196060 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:32.215337 14457 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.188:41835
I20260927 21:42:32.215355 14523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.188:41835 every 8 connection(s)
I20260927 21:42:32.216910 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
I20260927 21:42:32.220992 14524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:32.225227 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14457
I20260927 21:42:32.225432 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790545135802652-810-0/minicluster-data/master-2/wal/instance
I20260927 21:42:32.226037 810 external_mini_cluster.cc:1428] Setting key 7c4861d0e247d649092923ae595443a2
I20260927 21:42:32.229009 14524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:32.250406 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.233879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.0.202.190:33921)
0927 21:42:32.234406 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.234414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.235058 (+ 644us) server_negotiation.cc:407] Connection header received
0927 21:42:32.235266 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.235271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.235350 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.235462 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.236801 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.237540 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.239247 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.239759 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.239858 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.243940 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.243945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.243949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.243982 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.246596 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.247383 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.247389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.247395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.247528 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.247963 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.247970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.247973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.248263 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.248442 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.249486 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.249785 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":285,"threads_started":1}
I20260927 21:42:32.251291 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.228115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.0.202.188:41835)
0927 21:42:32.229037 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.229049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.229090 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:42:32.231366 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.231393 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.231921 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.232380 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.234048 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.235952 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.237588 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.238292 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.238417 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.243539 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.243554 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.243571 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.243619 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.246768 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.247667 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.247677 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.247681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.247917 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.248439 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.248443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.248446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.249031 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.249340 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.249930 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.250161 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":707,"spinlock_wait_cycles":20352,"thread_start_us":505,"threads_started":1}
I20260927 21:42:32.251271 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.233799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33921 (local address 127.0.0.1:48356)
0927 21:42:32.234919 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.234960 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:32.235139 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:32.235537 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:32.235546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:32.236260 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:32.236626 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.236640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.237708 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.237711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.239055 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.239067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.240011 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.240020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.240207 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.240842 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:32.240879 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:32.243732 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:32.246748 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.246759 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.246773 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.247253 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.247644 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.247649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.247653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.247850 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.248430 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:32.248438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:32.249034 (+ 596us) client_negotiation.cc:769] Sending connection context
0927 21:42:32.249334 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 21:42:32.249809 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":54,"thread_start_us":698,"threads_started":1}
I20260927 21:42:32.253012 14524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:32.268935 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.253694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:41489 (local address 127.0.0.1:44772)
0927 21:42:32.253899 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.253920 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:32.254024 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:32.254339 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:32.254343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:32.254829 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:32.255017 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.255030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.256464 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.256469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.257791 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.257806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.258966 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.258972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.259142 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.259829 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:32.259852 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:32.262537 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:32.266240 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.266244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.266247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.266809 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.267259 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.267262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.267264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.267401 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.268059 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:32.268063 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:32.268267 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 21:42:32.268460 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 21:42:32.268690 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 21:42:32.269212 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.253811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.0.202.189:41489)
0927 21:42:32.254014 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.254020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.254040 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:42:32.254139 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.254146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.254212 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.254318 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.255163 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.256275 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.258019 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.258787 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.258927 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.262770 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.262775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.262779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.262817 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.266027 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.266956 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.266961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.266965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.267128 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.267511 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.267517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.267521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.267923 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.268083 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.268610 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.268831 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:42:32.270186 14524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:42:32.272782 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.254768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.202.190:33921)
0927 21:42:32.254952 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.254958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.254980 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:42:32.255094 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.255102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.255195 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.255265 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.256824 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.257865 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.259716 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.260391 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.260530 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.265977 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.265984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.265988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.266042 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.269547 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.270470 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.270476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.270478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.270629 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.271211 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.271216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.271220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.271549 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.271759 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.272172 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.272465 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:42:32.284559 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.271101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.0.202.188:41835)
0927 21:42:32.271265 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.271270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.271285 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:42:32.271328 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.271331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.271378 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.271476 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.272333 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.273341 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.274925 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.275504 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.275602 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.278961 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.278965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.278968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.278995 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.281903 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.282536 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.282544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.282547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.282694 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.283069 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.283074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.283077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.283443 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.283608 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.284066 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.284239 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:42:32.285027 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.270901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54906)
0927 21:42:32.271057 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.271076 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:32.271169 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:32.271489 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:32.271493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:32.271990 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:32.272155 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.272167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.273480 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.273485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.274773 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.274790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.275687 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.275726 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.275967 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.276605 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:32.276630 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:32.278774 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:32.282059 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.282065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.282069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.282419 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.282814 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.282818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.282821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.282951 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.283588 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:32.283594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:32.283786 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 21:42:32.283983 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 21:42:32.284499 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":109}
I20260927 21:42:32.290690 14524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: Bootstrap starting.
I20260927 21:42:32.290802 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.274644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.0.202.189:41489)
0927 21:42:32.274798 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.274802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.274817 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:42:32.274968 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.274973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.275046 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.275227 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.276818 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.277588 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.278993 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.279401 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.279532 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.284741 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.284749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.284755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.284798 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.287967 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.288841 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.288845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.288848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.289000 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.289432 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.289435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.289437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.289727 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.289939 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.290294 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.290476 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:42:32.294368 14524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:32.295724 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:42:32.295722 14524 log.cc:824] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:32.298804 14524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: No bootstrap required, opened a new log
I20260927 21:42:32.299710 14376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 4
I20260927 21:42:32.308683 14524 raft_consensus.cc:348] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
I20260927 21:42:32.311589 14524 raft_consensus.cc:374] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:32.311754 14524 raft_consensus.cc:729] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b838d3810c9f4a16b2719ee6cab25fdc, State: Initialized, Role: FOLLOWER
I20260927 21:42:32.313594 14524 consensus_queue.cc:260] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.319222 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.300381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54920)
0927 21:42:32.300551 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.300571 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:32.300679 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:32.301080 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:32.301088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:32.301741 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:32.302041 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.302059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.303607 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.303616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.305531 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.305555 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.306688 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.306694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.306910 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.307679 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:32.307702 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:32.310583 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:32.316035 (+ 5452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.316040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.316044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.316653 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.317132 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.317137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.317140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.317338 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.318138 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:32.318145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:32.318338 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 21:42:32.318465 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:42:32.318777 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 21:42:32.320493 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.300442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.0.202.188:41835)
0927 21:42:32.300627 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.300633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.300667 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:42:32.300806 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.300812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.300894 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.301007 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.302235 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.303433 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.305725 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.306492 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.307239 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.310831 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.310839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.310845 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.310888 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.315864 (+ 4976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.316836 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.316841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.316844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.317004 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.317497 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.317505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.317508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.317993 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.319669 (+ 1676us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.319815 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.320069 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:42:32.328039 14376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0: Bootstrap starting.
I20260927 21:42:32.330749 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.301366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.0.202.190:33921)
0927 21:42:32.301552 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.301558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.309622 (+ 8064us) server_negotiation.cc:407] Connection header received
0927 21:42:32.309901 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.309906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.309982 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.310050 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.312623 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.313720 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.316035 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.316780 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.316873 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.320781 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.320786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.320791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.320837 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.326225 (+ 5388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.327198 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.327208 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.327214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.327360 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.327821 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.327826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.327831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.328370 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.328667 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.329872 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.330178 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:42:32.333751 14376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:32.334482 14538 sys_catalog.cc:455] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.335203 14376 log.cc:824] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:32.335443 14538 sys_catalog.cc:458] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:42:32.336020 14524 sys_catalog.cc:564] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:32.343562 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.306171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.0.202.189:41489)
0927 21:42:32.307002 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.307011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.310879 (+ 3868us) server_negotiation.cc:407] Connection header received
0927 21:42:32.311454 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.311462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.312041 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.312142 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.315983 (+ 3841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.317373 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.320406 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.321274 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.321394 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.335969 (+ 14575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.335985 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.335990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.336035 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.339973 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.340875 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.340881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.340885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.341067 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.341565 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.341573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.341577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.341978 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.342242 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.342922 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.343180 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644}
I20260927 21:42:32.345528 14376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0: No bootstrap required, opened a new log
I20260927 21:42:32.347379 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.309931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.0.202.188:41835)
0927 21:42:32.323016 (+ 13085us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.323022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.325909 (+ 2887us) server_negotiation.cc:407] Connection header received
0927 21:42:32.327258 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.327265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.327369 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.327438 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.331022 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.333391 (+ 2369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.335835 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.336553 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.336671 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.340346 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.340354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.340360 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.340400 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.344164 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.345009 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.345014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.345018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.345153 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.345636 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.345643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.345647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.346057 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.346289 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.346855 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.347079 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12971,"thread_start_us":20666,"threads_started":1}
I20260927 21:42:32.359404 14376 raft_consensus.cc:348] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.359787 14376 raft_consensus.cc:374] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:32.359887 14376 raft_consensus.cc:729] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23aacaa95474edaad33311a22e0fcf0, State: Initialized, Role: FOLLOWER
I20260927 21:42:32.360666 14376 consensus_queue.cc:260] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.366140 14553 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
W20260927 21:42:32.369691 14551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:42:32.369846 14551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:42:32.367528 14376 sys_catalog.cc:564] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:32.370041 14553 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:42:32.360778 14457 master_runner.cc:186] Error getting master registration for 127.0.202.190:33921: OK, instance_id { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" instance_seqno: 1790545351150135 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:42:32.373143 14457 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:42:32.373944 14553 raft_consensus.cc:482] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:42:32.374297 14553 raft_consensus.cc:504] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.376209 14553 leader_election.cc:288] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6cff6704c441e9aa161190b6515e32 (127.0.202.189:41489), b838d3810c9f4a16b2719ee6cab25fdc (127.0.202.188:41835)
I20260927 21:42:32.377054 14499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d23aacaa95474edaad33311a22e0fcf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" is_pre_election: true
I20260927 21:42:32.377677 14499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d23aacaa95474edaad33311a22e0fcf0 in term 0.
I20260927 21:42:32.378321 14325 leader_election.cc:302] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b838d3810c9f4a16b2719ee6cab25fdc, d23aacaa95474edaad33311a22e0fcf0; no voters:
I20260927 21:42:32.378778 14553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:42:32.378968 14553 raft_consensus.cc:482] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:42:32.379091 14553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:32.383482 14553 raft_consensus.cc:504] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.385465 14499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d23aacaa95474edaad33311a22e0fcf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b838d3810c9f4a16b2719ee6cab25fdc"
I20260927 21:42:32.385737 14499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:42:32.388442 14330 tablet.cc:2369] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:42:32.390525 14499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23aacaa95474edaad33311a22e0fcf0 in term 1.
I20260927 21:42:32.391232 14325 leader_election.cc:302] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b838d3810c9f4a16b2719ee6cab25fdc, d23aacaa95474edaad33311a22e0fcf0; no voters:
I20260927 21:42:32.393458 14564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:42:32.392813 14553 leader_election.cc:288] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [CANDIDATE]: Term 1 election: Requested vote from peers 6a6cff6704c441e9aa161190b6515e32 (127.0.202.189:41489), b838d3810c9f4a16b2719ee6cab25fdc (127.0.202.188:41835)
I20260927 21:42:32.400497 14564 raft_consensus.cc:686] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [term 1 LEADER]: Becoming Leader. State: Replica: d23aacaa95474edaad33311a22e0fcf0, State: Running, Role: LEADER
I20260927 21:42:32.401165 14564 consensus_queue.cc:237] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.404879 14553 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23aacaa95474edaad33311a22e0fcf0. Latest consensus state: current_term: 1 leader_uuid: "d23aacaa95474edaad33311a22e0fcf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.405334 14553 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:32.413671 14450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } attempt: 5
W20260927 21:42:32.413862 14565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 21:42:32.412811 14309 master_runner.cc:186] Error getting master registration for 127.0.202.189:41489: OK, instance_id { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" instance_seqno: 1790545351668396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:42:32.424005 14309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:42:32.424459 14565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 21:42:32 dist-test-slave-25dq krb5kdc[14296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
I20260927 21:42:32.427424 14569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:42:32.429638 14569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:42:32.437110 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.414234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:41835 (local address 127.0.0.1:54948)
0927 21:42:32.414607 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:32.414628 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:32.414737 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:32.416529 (+ 1792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:32.416535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:32.417101 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:32.417325 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.417338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.419085 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.419091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.420651 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:32.420667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.421873 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.421880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:32.422098 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.422828 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:32.422853 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:32.426338 (+ 3485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:32.434174 (+ 7836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.434179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.434183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.434772 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.435187 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:32.435191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:32.435194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:32.435383 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:32.436071 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:32.436089 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:32.436254 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:42:32.436421 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 21:42:32.436731 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260927 21:42:32.438935 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:32.415168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.0.202.188:41835)
0927 21:42:32.415890 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:32.415896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:32.415915 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:42:32.415969 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:32.415978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:32.416063 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:32.416171 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:32.417519 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.418640 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.420830 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:32.421514 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:32.421646 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:32.426952 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:32.426960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:32.426966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:32.427014 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:32.434017 (+ 7003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.434906 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.434912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.434915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.435073 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:32.435489 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:32.435494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:32.435498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:32.435886 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:32.436098 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:32.437635 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0927 21:42:32.438128 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603}
I20260927 21:42:32.440060 14499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [term 1 FOLLOWER]: Refusing update from remote peer d23aacaa95474edaad33311a22e0fcf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:32.441321 14553 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:32.444877 14450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32: Bootstrap starting.
W20260927 21:42:32.448258 14478 tablet.cc:2369] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:42:32.452273 14450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:32.455391 14450 log.cc:824] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:32.460906 14538 sys_catalog.cc:455] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23aacaa95474edaad33311a22e0fcf0. Latest consensus state: current_term: 1 leader_uuid: "d23aacaa95474edaad33311a22e0fcf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.461215 14538 sys_catalog.cc:458] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:42:32.463274 14570 mvcc.cc:205] Tried to move back new op lower bound from 7334073763582214144 to 7334073763440046080. Current Snapshot: MvccSnapshot[applied={T|T < 7334073763582214144}]
I20260927 21:42:32.466401 14571 mvcc.cc:205] Tried to move back new op lower bound from 7334073763582214144 to 7334073763440046080. Current Snapshot: MvccSnapshot[applied={T|T < 7334073763582214144}]
I20260927 21:42:32.469086 14538 sys_catalog.cc:455] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d23aacaa95474edaad33311a22e0fcf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.469393 14538 sys_catalog.cc:458] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:42:32.470625 14450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32: No bootstrap required, opened a new log
I20260927 21:42:32.477694 14566 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d23aacaa95474edaad33311a22e0fcf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.478135 14566 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:32.480747 14569 catalog_manager.cc:1347] Generated new cluster ID: c8052b680e1046f885089dfa02d55c81
I20260927 21:42:32.480880 14569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:42:32.501101 14450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.501646 14450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:32.501765 14450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6cff6704c441e9aa161190b6515e32, State: Initialized, Role: FOLLOWER
I20260927 21:42:32.502542 14450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } }
I20260927 21:42:32.503902 14576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.504781 14576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:42:32.512187 14425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d23aacaa95474edaad33311a22e0fcf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6cff6704c441e9aa161190b6515e32" is_pre_election: true
I20260927 21:42:32.512907 14425 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d23aacaa95474edaad33311a22e0fcf0 in term 0.
I20260927 21:42:32.513317 14424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d23aacaa95474edaad33311a22e0fcf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6cff6704c441e9aa161190b6515e32"
I20260927 21:42:32.513568 14424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:32.518028 14569 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:42:32.519640 14424 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23aacaa95474edaad33311a22e0fcf0 in term 1.
I20260927 21:42:32.519991 14569 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:42:32.520444 14423 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [term 1 FOLLOWER]: Refusing update from remote peer d23aacaa95474edaad33311a22e0fcf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:32.522369 14566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:32.522842 14450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:32.541121 14569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0: Generated new TSK 0
I20260927 21:42:32.542496 14569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:42:32.564640 14576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23aacaa95474edaad33311a22e0fcf0. Latest consensus state: current_term: 1 leader_uuid: "d23aacaa95474edaad33311a22e0fcf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.564667 14581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d23aacaa95474edaad33311a22e0fcf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.565126 14576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:42:32.565200 14581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:42:32.566365 14553 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d23aacaa95474edaad33311a22e0fcf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23aacaa95474edaad33311a22e0fcf0" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33921 } } peers { permanent_uuid: "6a6cff6704c441e9aa161190b6515e32" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 41489 } } peers { permanent_uuid: "b838d3810c9f4a16b2719ee6cab25fdc" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 41835 } } }
I20260927 21:42:32.566676 14553 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23aacaa95474edaad33311a22e0fcf0 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:32.601533 14595 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:42:33.372081 14551 catalog_manager.cc:1259] Loaded cluster ID: c8052b680e1046f885089dfa02d55c81
I20260927 21:42:33.372219 14551 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: loading cluster ID for follower catalog manager: success
I20260927 21:42:33.374728 14551 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: acquiring CA information for follower catalog manager: success
I20260927 21:42:33.376734 14551 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b838d3810c9f4a16b2719ee6cab25fdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:42:33.607818 14595 catalog_manager.cc:1259] Loaded cluster ID: c8052b680e1046f885089dfa02d55c81
I20260927 21:42:33.607923 14595 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32: loading cluster ID for follower catalog manager: success
I20260927 21:42:33.610899 14595 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32: acquiring CA information for follower catalog manager: success
I20260927 21:42:33.613068 14595 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a6cff6704c441e9aa161190b6515e32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:42:33.687630 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14309
I20260927 21:42:33.688974 14371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:33.803067 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14309
I20260927 21:42:33.824748 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14383
I20260927 21:42:33.826122 14445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:33.937415 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14383
I20260927 21:42:33.958412 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14457
I20260927 21:42:33.959594 14519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:34.071740 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14457
2026-09-27T21:42:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4751 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:42:34 dist-test-slave-25dq krb5kdc[14603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:42:34 dist-test-slave-25dq krb5kdc[14603](info): set up 2 sockets
Sep 27 21:42:34 dist-test-slave-25dq krb5kdc[14603](info): commencing operation
krb5kdc: starting...
W20260927 21:42:35.196928 810 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:42:35 dist-test-slave-25dq krb5kdc[14603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:42:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:42:35.449272 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:39137
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:39137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:35.782778 14616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:35.783098 14616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:35.783257 14616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:35.791195 14616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:35.791316 14616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:35.791405 14616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:35.791488 14616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:35.791571 14616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:35.801846 14616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:39137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:39137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:35.803256 14616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:35.804656 14616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:35.816977 14622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:35.817003 14621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:35.817538 14624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:35.817967 14616 server_base.cc:1034] running on GCE node
I20260927 21:42:35.818778 14616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:35.820207 14616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:35.821509 14616 hybrid_clock.cc:648] HybridClock initialized: now 1790545355821394 us; error 50 us; skew 500 ppm
I20260927 21:42:35.821874 14616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:35 dist-test-slave-25dq krb5kdc[14603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:35.825117 14616 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:42:35.826674 14616 webserver.cc:460] Webserver started at http://127.0.202.190:34551/ using document root <none> and password file <none>
I20260927 21:42:35.827383 14616 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:35.827530 14616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:35.827843 14616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:35.830935 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "9939f11444ce40759daf3d8f6d20e8e2"
format_stamp: "Formatted at 2026-09-27 21:42:35 on dist-test-slave-25dq"
server_key: "8b926fa00092a7a134a9cf7e618d57d3"
server_key_iv: "b93c8839953ef631136b10c32ad31183"
server_key_version: "encryptionkey@0"
I20260927 21:42:35.831763 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "9939f11444ce40759daf3d8f6d20e8e2"
format_stamp: "Formatted at 2026-09-27 21:42:35 on dist-test-slave-25dq"
server_key: "8b926fa00092a7a134a9cf7e618d57d3"
server_key_iv: "b93c8839953ef631136b10c32ad31183"
server_key_version: "encryptionkey@0"
I20260927 21:42:35.837219 14616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:42:35.840948 14631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:35.842386 14616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:42:35.842561 14616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "9939f11444ce40759daf3d8f6d20e8e2"
format_stamp: "Formatted at 2026-09-27 21:42:35 on dist-test-slave-25dq"
server_key: "8b926fa00092a7a134a9cf7e618d57d3"
server_key_iv: "b93c8839953ef631136b10c32ad31183"
server_key_version: "encryptionkey@0"
I20260927 21:42:35.842767 14616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:35.860684 14616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:35.865054 14616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:35.865342 14616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:35.884964 14616 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:39137
I20260927 21:42:35.884994 14682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:39137 every 8 connection(s)
I20260927 21:42:35.886469 14616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:42:35.891076 14683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:35.893239 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14616
I20260927 21:42:35.893505 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:42:35.894192 810 external_mini_cluster.cc:1428] Setting key a1b8458a2ab88d8b1e83e5544ba77df9
I20260927 21:42:35.908583 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2: Bootstrap starting.
Sep 27 21:42:35 dist-test-slave-25dq krb5kdc[14603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:35.911744 14683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:35.912889 14683 log.cc:824] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:35.915846 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2: No bootstrap required, opened a new log
I20260927 21:42:35.919456 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:35.896214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.0.202.190:39137)
0927 21:42:35.897258 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:35.897277 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:35.897337 (+ 60us) server_negotiation.cc:407] Connection header received
0927 21:42:35.899436 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:35.899473 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:35.900130 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:35.900686 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:35.902234 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:35.904765 (+ 2531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:35.906259 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:35.907366 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:35.907549 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:35.912024 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:35.912039 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:35.912059 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:35.912124 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:35.915280 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:35.916014 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:35.916028 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:35.916034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:35.916203 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:35.916639 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:35.916646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:35.916650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:35.917189 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:35.917500 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:35.917974 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 21:42:35.918434 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":433,"threads_started":1}
I20260927 21:42:35.923415 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:42:35.924396 14683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9939f11444ce40759daf3d8f6d20e8e2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39137 } }
I20260927 21:42:35.924780 14683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:35.924883 14683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9939f11444ce40759daf3d8f6d20e8e2, State: Initialized, Role: FOLLOWER
I20260927 21:42:35.925434 14683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9939f11444ce40759daf3d8f6d20e8e2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39137 } }
I20260927 21:42:35.925680 14683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:42:35.925822 14683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:42:35.926014 14683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:35.928668 14683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9939f11444ce40759daf3d8f6d20e8e2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39137 } }
I20260927 21:42:35.929190 14683 leader_election.cc:302] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9939f11444ce40759daf3d8f6d20e8e2; no voters:
I20260927 21:42:35.929790 14683 leader_election.cc:288] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:42:35.930037 14688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:42:35.930792 14688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 9939f11444ce40759daf3d8f6d20e8e2, State: Running, Role: LEADER
I20260927 21:42:35.931317 14688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9939f11444ce40759daf3d8f6d20e8e2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39137 } }
I20260927 21:42:35.931967 14683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:35.934248 14690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9939f11444ce40759daf3d8f6d20e8e2. Latest consensus state: current_term: 1 leader_uuid: "9939f11444ce40759daf3d8f6d20e8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9939f11444ce40759daf3d8f6d20e8e2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39137 } } }
I20260927 21:42:35.934211 14689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9939f11444ce40759daf3d8f6d20e8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9939f11444ce40759daf3d8f6d20e8e2" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39137 } } }
I20260927 21:42:35.934628 14690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:35.934676 14689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:35.941869 14694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:42:35.947670 14694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:42:35.956817 14694 catalog_manager.cc:1347] Generated new cluster ID: 6e6ff91d56944846b34471fd8543ef3a
I20260927 21:42:35.956952 14694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:42:35.973512 14694 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:42:35.975947 14694 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:42:35.987074 14694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9939f11444ce40759daf3d8f6d20e8e2: Generated new TSK 0
I20260927 21:42:35.988370 14694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:42:36.037439 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14616
I20260927 21:42:36.038772 14678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:36.139168 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14616
2026-09-27T21:42:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2058 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:42:36 dist-test-slave-25dq krb5kdc[14711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:42:36 dist-test-slave-25dq krb5kdc[14711](info): set up 2 sockets
Sep 27 21:42:36 dist-test-slave-25dq krb5kdc[14711](info): commencing operation
krb5kdc: starting...
W20260927 21:42:37.288940 810 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:42:37 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:42:37.540738 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35289
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:37.874011 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:37.874310 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:37.874516 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:37.881891 14724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:37.881997 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:37.882102 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:37.882202 14724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:37.882272 14724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:37.892429 14724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:35289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:37.893986 14724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:37.895349 14724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:37.907812 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:37.907907 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:37.907814 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:37.908862 14724 server_base.cc:1034] running on GCE node
I20260927 21:42:37.909672 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:37.911145 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:37.912411 14724 hybrid_clock.cc:648] HybridClock initialized: now 1790545357912390 us; error 93 us; skew 500 ppm
I20260927 21:42:37.912784 14724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:37 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:37.916096 14724 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:42:37.917814 14724 webserver.cc:460] Webserver started at http://127.0.202.190:46433/ using document root <none> and password file <none>
I20260927 21:42:37.918517 14724 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:37.918668 14724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:37.918983 14724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:37.922060 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "ed8c7d09c0474daa85dd1581b092cf38"
format_stamp: "Formatted at 2026-09-27 21:42:37 on dist-test-slave-25dq"
server_key: "62aca94e136ecbac1e6387f499eeb6c0"
server_key_iv: "3c80a0d6ead12e931b59fcfa0ea095d6"
server_key_version: "encryptionkey@0"
I20260927 21:42:37.922830 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "ed8c7d09c0474daa85dd1581b092cf38"
format_stamp: "Formatted at 2026-09-27 21:42:37 on dist-test-slave-25dq"
server_key: "62aca94e136ecbac1e6387f499eeb6c0"
server_key_iv: "3c80a0d6ead12e931b59fcfa0ea095d6"
server_key_version: "encryptionkey@0"
I20260927 21:42:37.928184 14724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:42:37.931773 14739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:37.933218 14724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:42:37.933403 14724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ed8c7d09c0474daa85dd1581b092cf38"
format_stamp: "Formatted at 2026-09-27 21:42:37 on dist-test-slave-25dq"
server_key: "62aca94e136ecbac1e6387f499eeb6c0"
server_key_iv: "3c80a0d6ead12e931b59fcfa0ea095d6"
server_key_version: "encryptionkey@0"
I20260927 21:42:37.933604 14724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:37.959054 14724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:37.963791 14724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:37.964089 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:37.983794 14724 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:35289
I20260927 21:42:37.983891 14790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:35289 every 8 connection(s)
I20260927 21:42:37.985245 14724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:42:37.989178 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:37.995313 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14724
I20260927 21:42:37.995563 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:42:37.996403 810 external_mini_cluster.cc:1428] Setting key 488683643944e1863449addeb3c49cea
I20260927 21:42:38.001055 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Bootstrap starting.
I20260927 21:42:38.004163 14791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:38.005286 14791 log.cc:824] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:38.008528 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: No bootstrap required, opened a new log
Sep 27 21:42:38 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:38.016484 14791 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:38.016921 14791 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:38.017027 14791 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8c7d09c0474daa85dd1581b092cf38, State: Initialized, Role: FOLLOWER
I20260927 21:42:38.017728 14791 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:38.017978 14791 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:42:38.018138 14791 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:42:38.018362 14791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:38.021833 14791 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:38.022442 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:37.998684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.202.190:35289)
0927 21:42:37.999596 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:37.999617 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:37.999662 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:42:38.001626 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:38.001651 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:38.002056 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:38.002483 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:38.004062 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:38.005626 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:38.007606 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:38.008424 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:38.008613 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:38.013329 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:38.013341 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:38.013362 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:38.013409 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:38.017274 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:38.018064 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:38.018072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:38.018081 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:38.018335 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:38.018937 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:38.018943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:38.018949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:38.019729 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:38.020072 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:38.020468 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 21:42:38.021038 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":385,"threads_started":1}
I20260927 21:42:38.022475 14791 leader_election.cc:302] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8c7d09c0474daa85dd1581b092cf38; no voters:
I20260927 21:42:38.023167 14791 leader_election.cc:288] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:42:38.023375 14796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:42:38.024724 14796 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 LEADER]: Becoming Leader. State: Replica: ed8c7d09c0474daa85dd1581b092cf38, State: Running, Role: LEADER
I20260927 21:42:38.025475 14791 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:38.025395 14796 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:38.029057 14798 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8c7d09c0474daa85dd1581b092cf38. Latest consensus state: current_term: 1 leader_uuid: "ed8c7d09c0474daa85dd1581b092cf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } } }
I20260927 21:42:38.029444 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed8c7d09c0474daa85dd1581b092cf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } } }
I20260927 21:42:38.030136 14798 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:38.030141 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:38.032111 14804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:42:38.038130 14804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:42:38.047896 14804 catalog_manager.cc:1347] Generated new cluster ID: c98adfafe612403eb3860afe0ed9b174
I20260927 21:42:38.048033 14804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:42:38.073072 14804 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:42:38.074868 14804 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:42:38.092135 14804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Generated new TSK 0
I20260927 21:42:38.093235 14804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:42:38.167845 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35289
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:38.514245 14817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:38.514616 14817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:38.514784 14817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:38.522541 14817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:38.522704 14817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:38.522886 14817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:42:38.533638 14817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:38.534996 14817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:38.536378 14817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:38.549090 14822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:38.550086 14823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:38.551234 14817 server_base.cc:1034] running on GCE node
W20260927 21:42:38.551496 14825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:38.552238 14817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:38.553437 14817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:38.554672 14817 hybrid_clock.cc:648] HybridClock initialized: now 1790545358554620 us; error 36 us; skew 500 ppm
I20260927 21:42:38.555074 14817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:38 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:38.558609 14817 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:42:38.560271 14817 webserver.cc:460] Webserver started at http://127.0.202.129:33819/ using document root <none> and password file <none>
I20260927 21:42:38.561061 14817 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:38.561209 14817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:38.561525 14817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:38.564630 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "cc889ac6795243d98ea72543575f8f80"
format_stamp: "Formatted at 2026-09-27 21:42:38 on dist-test-slave-25dq"
server_key: "0979ce6e1791f58f5799667c129674de"
server_key_iv: "d222c89d34b2894c023952daf905cdb1"
server_key_version: "encryptionkey@0"
I20260927 21:42:38.565443 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "cc889ac6795243d98ea72543575f8f80"
format_stamp: "Formatted at 2026-09-27 21:42:38 on dist-test-slave-25dq"
server_key: "0979ce6e1791f58f5799667c129674de"
server_key_iv: "d222c89d34b2894c023952daf905cdb1"
server_key_version: "encryptionkey@0"
I20260927 21:42:38.570849 14817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:42:38.574429 14832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:38.575938 14817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:42:38.576133 14817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "cc889ac6795243d98ea72543575f8f80"
format_stamp: "Formatted at 2026-09-27 21:42:38 on dist-test-slave-25dq"
server_key: "0979ce6e1791f58f5799667c129674de"
server_key_iv: "d222c89d34b2894c023952daf905cdb1"
server_key_version: "encryptionkey@0"
I20260927 21:42:38.576392 14817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:38.597146 14817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:38.601604 14817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:38.601910 14817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:38.602943 14817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:38.604552 14817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:38.604653 14817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:38.604803 14817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:38.604874 14817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:38.653570 14817 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41717
I20260927 21:42:38.653630 14944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41717 every 8 connection(s)
I20260927 21:42:38.654894 14817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:42:38.664878 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14817
I20260927 21:42:38.665095 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:42:38.665802 810 external_mini_cluster.cc:1428] Setting key 2353e4443dbbdfa57db34c5638bc5ef4
Sep 27 21:42:38 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:38.677839 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:38.657706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:59101 (local address 127.0.202.190:35289)
0927 21:42:38.658209 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:38.658214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:38.659158 (+ 944us) server_negotiation.cc:407] Connection header received
0927 21:42:38.661152 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:38.661158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:38.661254 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:38.661380 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:38.662927 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:38.663651 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:38.665387 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:38.665795 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:38.665936 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:38.669880 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:38.669889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:38.669897 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:38.669944 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:38.673808 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:38.674875 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:38.674884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:38.674891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:38.675042 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:38.675654 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:38.675662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:38.675669 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:38.676050 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:38.676171 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:38.677361 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0927 21:42:38.677544 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
I20260927 21:42:38.679566 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:38.658268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:59101)
0927 21:42:38.658961 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:38.659031 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:38.660875 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:38.661622 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:38.661631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:38.662152 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:38.662751 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:38.662768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:38.663810 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:38.663814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:38.665241 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:38.665254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:38.666030 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:38.666038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:38.666275 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:38.666971 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:38.667009 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:38.669645 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:38.673975 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:38.673988 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:38.674011 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:38.674715 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:38.675175 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:38.675183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:38.675189 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:38.675509 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:38.676204 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:38.676215 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:38.676795 (+ 580us) client_negotiation.cc:769] Sending connection context
0927 21:42:38.677209 (+ 414us) client_negotiation.cc:240] Negotiation successful
0927 21:42:38.677887 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":257,"threads_started":1}
I20260927 21:42:38.681672 14945 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35289
I20260927 21:42:38.682046 14945 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:38.683079 14945 heartbeater.cc:511] Master 127.0.202.190:35289 requested a full tablet report, sending...
I20260927 21:42:38.686077 14756 ts_manager.cc:194] Registered new tserver with Master: cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717)
I20260927 21:42:38.688741 14756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:59101
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:42:38.729396 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35289
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:39.070173 14951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:39.070525 14951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:39.070693 14951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:39.078467 14951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:39.078642 14951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:39.078828 14951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:42:39.089668 14951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:39.091215 14951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:39.092661 14951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:39.107136 14956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:39.107208 14959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:39.107088 14957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:39.108057 14951 server_base.cc:1034] running on GCE node
I20260927 21:42:39.108733 14951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:39.109702 14951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:39.110934 14951 hybrid_clock.cc:648] HybridClock initialized: now 1790545359110864 us; error 38 us; skew 500 ppm
I20260927 21:42:39.111227 14951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:39 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:39.115018 14951 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:42:39.116717 14951 webserver.cc:460] Webserver started at http://127.0.202.130:43909/ using document root <none> and password file <none>
I20260927 21:42:39.117508 14951 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:39.117666 14951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:39.117986 14951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:39.121249 14951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "da00c0408b0b4856b99c18f4f1d325a2"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "045f9ae191ba10176576fc5230090e9f"
server_key_iv: "6ea68bdd0855f1ae0c179f3cdb879f05"
server_key_version: "encryptionkey@0"
I20260927 21:42:39.122061 14951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "da00c0408b0b4856b99c18f4f1d325a2"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "045f9ae191ba10176576fc5230090e9f"
server_key_iv: "6ea68bdd0855f1ae0c179f3cdb879f05"
server_key_version: "encryptionkey@0"
I20260927 21:42:39.129089 14951 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 21:42:39.133478 14966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:39.135051 14951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:42:39.135262 14951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "da00c0408b0b4856b99c18f4f1d325a2"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "045f9ae191ba10176576fc5230090e9f"
server_key_iv: "6ea68bdd0855f1ae0c179f3cdb879f05"
server_key_version: "encryptionkey@0"
I20260927 21:42:39.135488 14951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:39.175125 14951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:39.180647 14951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:39.180972 14951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:39.182070 14951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:39.183542 14951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:39.183652 14951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:39.183820 14951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:39.183895 14951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:39.222540 14951 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:41101
I20260927 21:42:39.222628 15078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:41101 every 8 connection(s)
I20260927 21:42:39.224023 14951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:42:39.225981 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 14951
I20260927 21:42:39.226178 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:42:39.226797 810 external_mini_cluster.cc:1428] Setting key 2e75b0cbbb903a3d4f5cd6781a2324b5
Sep 27 21:42:39 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:39.247628 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.227124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:40353 (local address 127.0.202.190:35289)
0927 21:42:39.227698 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.227706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.229001 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 21:42:39.230992 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.230998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.231089 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.231210 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:39.232943 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.233870 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.235587 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.236064 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.236207 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.241140 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:39.241146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:39.241150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:39.241189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:39.244133 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:39.245084 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:39.245089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:39.245092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:39.245230 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:39.245707 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:39.245712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:39.245715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:39.246011 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:39.246179 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.246935 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.247209 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":287,"threads_started":1}
I20260927 21:42:39.248741 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.227884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:40353)
0927 21:42:39.228806 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:39.228861 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:39.230681 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:39.231473 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:39.231482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:39.232042 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:39.232768 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.232789 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.234030 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.234035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.235426 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.235439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.236341 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.236349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.236774 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.237625 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:39.237668 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:39.240873 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:39.244326 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:39.244337 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:39.244356 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:39.244930 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:39.245359 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:39.245366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:39.245370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:39.245575 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:39.246146 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:39.246155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:39.246546 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 21:42:39.246831 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 21:42:39.247429 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":399,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
I20260927 21:42:39.250445 15079 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35289
I20260927 21:42:39.250761 15079 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:39.251631 15079 heartbeater.cc:511] Master 127.0.202.190:35289 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
I20260927 21:42:39.253544 14756 ts_manager.cc:194] Registered new tserver with Master: da00c0408b0b4856b99c18f4f1d325a2 (127.0.202.130:41101)
I20260927 21:42:39.254487 14756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:40353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:42:39.283105 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35289
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:39.614732 15085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:39.615023 15085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:39.615186 15085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:39.622457 15085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:39.622588 15085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:39.622776 15085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:42:39.633281 15085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:39.634718 15085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:39.636098 15085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:39.650187 15090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:39.650187 15091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:39.650219 15093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:39.651218 15085 server_base.cc:1034] running on GCE node
I20260927 21:42:39.651971 15085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:39.653218 15085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:39.654556 15085 hybrid_clock.cc:648] HybridClock initialized: now 1790545359654432 us; error 57 us; skew 500 ppm
I20260927 21:42:39.654939 15085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:39 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:39.658603 15085 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:42:39.660216 15085 webserver.cc:460] Webserver started at http://127.0.202.131:45527/ using document root <none> and password file <none>
I20260927 21:42:39.660954 15085 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:39.661098 15085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:39.661405 15085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:42:39.664535 15085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "daba5937ef12466b82364ab43f46d203"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "520941fd3085c899c990d9f88364e0d4"
server_key_iv: "28dcba70c7b979d2b26bc143e0a304c7"
server_key_version: "encryptionkey@0"
I20260927 21:42:39.665373 15085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "daba5937ef12466b82364ab43f46d203"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "520941fd3085c899c990d9f88364e0d4"
server_key_iv: "28dcba70c7b979d2b26bc143e0a304c7"
server_key_version: "encryptionkey@0"
I20260927 21:42:39.670933 15085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:42:39.674696 15100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:39.676147 15085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:42:39.676414 15085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "daba5937ef12466b82364ab43f46d203"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "520941fd3085c899c990d9f88364e0d4"
server_key_iv: "28dcba70c7b979d2b26bc143e0a304c7"
server_key_version: "encryptionkey@0"
I20260927 21:42:39.676644 15085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:39.693403 14945 heartbeater.cc:503] Master 127.0.202.190:35289 was elected leader, sending a full tablet report...
I20260927 21:42:39.699357 15085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:39.703742 15085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:39.704039 15085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:39.705101 15085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:39.706607 15085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:39.706687 15085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:39.706774 15085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:39.706818 15085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:39.742683 15085 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:43879
I20260927 21:42:39.742723 15212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:43879 every 8 connection(s)
I20260927 21:42:39.744408 15085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:42:39.747377 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 15085
I20260927 21:42:39.747611 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:42:39.748137 810 external_mini_cluster.cc:1428] Setting key 78236bd71aafe2b3e3baf3d2a94ecafe
Sep 27 21:42:39 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:39.767047 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.747472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:35117 (local address 127.0.202.190:35289)
0927 21:42:39.747742 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.747749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.749225 (+ 1476us) server_negotiation.cc:407] Connection header received
0927 21:42:39.751888 (+ 2663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.751895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.751993 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.752096 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:39.753881 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.754639 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.756275 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.756755 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.756851 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.760385 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:39.760390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:39.760400 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:39.760433 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:39.763651 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:39.764653 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:39.764658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:39.764662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:39.764829 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:39.765330 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:39.765334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:39.765337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:39.765632 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:39.765776 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.766583 (+ 807us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.766784 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 21:42:39.768199 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.748006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:35117)
0927 21:42:39.749008 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:39.749068 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:39.751512 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:39.752419 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:39.752429 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:39.753068 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:39.753690 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.753711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.754818 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.754823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.756137 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.756150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.756968 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.756973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.757180 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.757899 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:39.757922 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:39.760124 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:39.763831 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:39.763839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:39.763886 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:39.764533 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:39.764951 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:39.764954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:39.764957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:39.765140 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:39.765800 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:39.765807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:39.766162 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 21:42:39.766450 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 21:42:39.766893 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":377,"threads_started":1}
I20260927 21:42:39.769757 15213 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35289
I20260927 21:42:39.770120 15213 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:39.770941 15213 heartbeater.cc:511] Master 127.0.202.190:35289 requested a full tablet report, sending...
I20260927 21:42:39.772423 14756 ts_manager.cc:194] Registered new tserver with Master: daba5937ef12466b82364ab43f46d203 (127.0.202.131:43879)
I20260927 21:42:39.773227 14756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:35117
I20260927 21:42:39.777384 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:42:39.795626 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.782215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.202.190:35289)
0927 21:42:39.782394 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.782399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.782841 (+ 442us) server_negotiation.cc:407] Connection header received
0927 21:42:39.782990 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.782993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.783055 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.783154 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:39.784543 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.785295 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.786739 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.787222 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.787406 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.789934 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:39.789939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:39.789942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:39.789976 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:39.792895 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:39.793668 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:39.793673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:39.793675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:39.793827 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:39.794176 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:39.794180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:39.794182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:39.794579 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:39.794769 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.795116 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.795361 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:42:39.800099 14756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:42:39.807184 14756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:42:39.831494 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.820623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.0.202.129:41717)
0927 21:42:39.822723 (+ 2100us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.822730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.822751 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:42:39.822827 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.822833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.823024 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.823956 (+ 932us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:39.824256 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.825811 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.828801 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.830301 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.830436 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.830686 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.830986 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.831162 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1949,"thread_start_us":1786,"threads_started":1}
I20260927 21:42:39.831489 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.820359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41717 (local address 127.0.0.1:34666)
0927 21:42:39.821044 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:39.821091 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:39.821254 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:39.823256 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:39.823264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:39.823294 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:39.823819 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.823834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.825974 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.825981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.828585 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.828602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.830512 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.830519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.830722 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.830773 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:42:39.830938 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 21:42:39.831128 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":307,"threads_started":1}
I20260927 21:42:39.832358 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.823326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:43879 (local address 127.0.0.1:42674)
0927 21:42:39.825204 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:39.825221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:39.825336 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:39.825904 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:39.825912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:39.825943 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:39.826500 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.826509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.827931 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.827936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.829914 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.829926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.831636 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.831643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.831831 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.831869 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:42:39.831941 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:42:39.832053 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"mutex_wait_us":28,"thread_start_us":1598,"threads_started":1}
I20260927 21:42:39.834332 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.821002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:41101 (local address 127.0.0.1:34864)
0927 21:42:39.821881 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:39.821899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:39.822018 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:39.823428 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:39.823433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:39.823458 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:39.826215 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.826227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.828251 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.828256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.830522 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.830536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.833535 (+ 2999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.833541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.833724 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.833755 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:42:39.833851 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:42:39.834000 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":1,"thread_start_us":603,"threads_started":1}
I20260927 21:42:39.834378 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.823458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.202.131:43879)
0927 21:42:39.824751 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.824760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.825413 (+ 653us) server_negotiation.cc:407] Connection header received
0927 21:42:39.825521 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.825527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.825731 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.825870 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:39.826660 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.827780 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.830071 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.831408 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.832431 (+ 1023us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.833707 (+ 1276us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.833773 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.833975 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1081,"thread_start_us":685,"threads_started":1}
I20260927 21:42:39.834442 15227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.820681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.0.202.130:41101)
0927 21:42:39.822547 (+ 1866us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.822557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.822589 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:42:39.822686 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.822696 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.822962 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.823167 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:39.826467 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.828084 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.830703 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.833328 (+ 2625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.833480 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.833669 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.833963 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.834132 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1675,"thread_start_us":1528,"threads_started":1}
I20260927 21:42:39.835909 14880 tablet_service.cc:1467] Processing CreateTablet for tablet da4e8eab257845b39bdb98c9c3e02e26 (DEFAULT_TABLE table=test-table [id=296881c1ff444bec9311529381981e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:39.837568 14880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4e8eab257845b39bdb98c9c3e02e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:39.838275 15148 tablet_service.cc:1467] Processing CreateTablet for tablet da4e8eab257845b39bdb98c9c3e02e26 (DEFAULT_TABLE table=test-table [id=296881c1ff444bec9311529381981e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:39.839143 15014 tablet_service.cc:1467] Processing CreateTablet for tablet da4e8eab257845b39bdb98c9c3e02e26 (DEFAULT_TABLE table=test-table [id=296881c1ff444bec9311529381981e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:39.839704 15148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4e8eab257845b39bdb98c9c3e02e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:39.840345 15014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4e8eab257845b39bdb98c9c3e02e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:39.852443 15231 tablet_bootstrap.cc:492] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Bootstrap starting.
I20260927 21:42:39.854562 15232 tablet_bootstrap.cc:492] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Bootstrap starting.
I20260927 21:42:39.856346 15233 tablet_bootstrap.cc:492] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Bootstrap starting.
I20260927 21:42:39.856442 15231 tablet_bootstrap.cc:654] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:39.857888 15231 log.cc:824] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:39.858296 15232 tablet_bootstrap.cc:654] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:39.859607 15233 tablet_bootstrap.cc:654] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:39.860018 15232 log.cc:824] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:39.860652 15233 log.cc:824] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:39.863485 15233 tablet_bootstrap.cc:492] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: No bootstrap required, opened a new log
I20260927 21:42:39.863591 15231 tablet_bootstrap.cc:492] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: No bootstrap required, opened a new log
I20260927 21:42:39.863591 15232 tablet_bootstrap.cc:492] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: No bootstrap required, opened a new log
I20260927 21:42:39.863806 15233 ts_tablet_manager.cc:1397] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 21:42:39.863873 15231 ts_tablet_manager.cc:1397] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 21:42:39.863874 15232 ts_tablet_manager.cc:1397] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 21:42:39.872088 15233 raft_consensus.cc:348] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.872596 15233 raft_consensus.cc:374] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:39.872737 15233 raft_consensus.cc:729] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daba5937ef12466b82364ab43f46d203, State: Initialized, Role: FOLLOWER
I20260927 21:42:39.873427 15233 consensus_queue.cc:260] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.874629 15213 heartbeater.cc:503] Master 127.0.202.190:35289 was elected leader, sending a full tablet report...
I20260927 21:42:39.875412 15233 ts_tablet_manager.cc:1428] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260927 21:42:39.875459 15231 raft_consensus.cc:348] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.875459 15232 raft_consensus.cc:348] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.875993 15232 raft_consensus.cc:374] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:39.876005 15231 raft_consensus.cc:374] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:39.876112 15231 raft_consensus.cc:729] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da00c0408b0b4856b99c18f4f1d325a2, State: Initialized, Role: FOLLOWER
I20260927 21:42:39.876150 15232 raft_consensus.cc:729] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc889ac6795243d98ea72543575f8f80, State: Initialized, Role: FOLLOWER
I20260927 21:42:39.876931 15231 consensus_queue.cc:260] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.876931 15232 consensus_queue.cc:260] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.879428 15232 ts_tablet_manager.cc:1428] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Time spent starting tablet: real 0.015s user 0.013s sys 0.005s
I20260927 21:42:39.881309 15079 heartbeater.cc:503] Master 127.0.202.190:35289 was elected leader, sending a full tablet report...
I20260927 21:42:39.882030 15231 ts_tablet_manager.cc:1428] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
W20260927 21:42:39.907431 14946 tablet.cc:2369] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:42:39.940194 15237 raft_consensus.cc:482] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:42:39.940544 15237 raft_consensus.cc:504] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.942224 15237 leader_election.cc:288] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717), da00c0408b0b4856b99c18f4f1d325a2 (127.0.202.130:41101)
I20260927 21:42:39.948707 15227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.942689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:58491 (local address 127.0.202.130:41101)
0927 21:42:39.942890 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.942896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.942917 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:42:39.943107 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.943112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.943172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.943265 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:39.943710 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.944736 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.946537 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.947677 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.947806 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.947859 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.948390 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.948512 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:42:39.948825 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.942499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:41101 (local address 127.0.202.131:58491)
0927 21:42:39.942789 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:39.942809 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:39.942938 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:39.943289 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:39.943297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:39.943332 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:39.943538 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.943549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.944864 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.944869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.946373 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.946385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.947847 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.947853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.948077 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.948119 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:42:39.948351 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 21:42:39.948513 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 21:42:39.949692 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.942574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:43239 (local address 127.0.202.129:41717)
0927 21:42:39.942804 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.942814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.943539 (+ 725us) server_negotiation.cc:407] Connection header received
0927 21:42:39.943679 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.943688 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.943745 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.943809 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:39.944425 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.945384 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.947416 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.948705 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.948896 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.948992 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.949366 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.949516 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:42:39.949796 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.942539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41717 (local address 127.0.202.131:43239)
0927 21:42:39.943434 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:39.943448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:39.943557 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:39.943852 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:39.943857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:39.943880 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:39.944241 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.944252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.945530 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.945533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.947249 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:39.947263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.949085 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.949091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:39.949216 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.949239 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:42:39.949314 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:42:39.949405 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":1,"thread_start_us":532,"threads_started":1}
I20260927 21:42:39.950057 15034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4e8eab257845b39bdb98c9c3e02e26" candidate_uuid: "daba5937ef12466b82364ab43f46d203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da00c0408b0b4856b99c18f4f1d325a2" is_pre_election: true
I20260927 21:42:39.950621 15034 raft_consensus.cc:2454] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daba5937ef12466b82364ab43f46d203 in term 0.
I20260927 21:42:39.950843 14900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4e8eab257845b39bdb98c9c3e02e26" candidate_uuid: "daba5937ef12466b82364ab43f46d203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc889ac6795243d98ea72543575f8f80" is_pre_election: true
I20260927 21:42:39.951378 14900 raft_consensus.cc:2454] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daba5937ef12466b82364ab43f46d203 in term 0.
I20260927 21:42:39.951416 15102 leader_election.cc:302] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da00c0408b0b4856b99c18f4f1d325a2, daba5937ef12466b82364ab43f46d203; no voters:
I20260927 21:42:39.951817 15237 raft_consensus.cc:2781] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:42:39.951988 15237 raft_consensus.cc:482] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:42:39.952109 15237 raft_consensus.cc:3037] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:39.954762 15237 raft_consensus.cc:504] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.955435 15237 leader_election.cc:288] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [CANDIDATE]: Term 1 election: Requested vote from peers cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717), da00c0408b0b4856b99c18f4f1d325a2 (127.0.202.130:41101)
I20260927 21:42:39.955881 14900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4e8eab257845b39bdb98c9c3e02e26" candidate_uuid: "daba5937ef12466b82364ab43f46d203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc889ac6795243d98ea72543575f8f80"
I20260927 21:42:39.956025 15034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4e8eab257845b39bdb98c9c3e02e26" candidate_uuid: "daba5937ef12466b82364ab43f46d203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da00c0408b0b4856b99c18f4f1d325a2"
I20260927 21:42:39.956120 14900 raft_consensus.cc:3037] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:39.956246 15034 raft_consensus.cc:3037] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:39.959474 14900 raft_consensus.cc:2454] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daba5937ef12466b82364ab43f46d203 in term 1.
I20260927 21:42:39.959474 15034 raft_consensus.cc:2454] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daba5937ef12466b82364ab43f46d203 in term 1.
I20260927 21:42:39.959944 15102 leader_election.cc:302] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da00c0408b0b4856b99c18f4f1d325a2, daba5937ef12466b82364ab43f46d203; no voters:
I20260927 21:42:39.960366 15237 raft_consensus.cc:2781] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:42:39.960815 15237 raft_consensus.cc:686] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 1 LEADER]: Becoming Leader. State: Replica: daba5937ef12466b82364ab43f46d203, State: Running, Role: LEADER
I20260927 21:42:39.961408 15237 consensus_queue.cc:237] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } }
I20260927 21:42:39.966188 14756 catalog_manager.cc:5495] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 reported cstate change: term changed from 0 to 1, leader changed from <none> to daba5937ef12466b82364ab43f46d203 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "daba5937ef12466b82364ab43f46d203" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } health_report { overall_health: HEALTHY } } }
W20260927 21:42:39.976240 15080 tablet.cc:2369] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:42:39.996245 15214 tablet.cc:2369] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:42:39.996713 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:39.990460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.202.131:43879)
0927 21:42:39.990648 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:39.990652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:39.990670 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:42:39.990832 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:39.990836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:39.990938 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:39.991060 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:42:39.991471 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.992181 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.994362 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:39.994750 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:39.994842 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:39.995413 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:42:39.995962 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:42:39.996147 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:39.996344 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 21:42:39.996439 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:42:40.005678 15034 raft_consensus.cc:1261] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 FOLLOWER]: Refusing update from remote peer daba5937ef12466b82364ab43f46d203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:40.005728 14900 raft_consensus.cc:1261] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Refusing update from remote peer daba5937ef12466b82364ab43f46d203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:40.006757 15241 consensus_queue.cc:1035] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:40.007189 15237 consensus_queue.cc:1035] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:40.019972 15244 mvcc.cc:205] Tried to move back new op lower bound from 7334073794572275712 to 7334073794406027264. Current Snapshot: MvccSnapshot[applied={T|T < 7334073794572275712}]
I20260927 21:42:40.021799 15245 mvcc.cc:205] Tried to move back new op lower bound from 7334073794572275712 to 7334073794406027264. Current Snapshot: MvccSnapshot[applied={T|T < 7334073794572275712}]
I20260927 21:42:40.033515 14755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:42:40.033890 14755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:42:40.037489 14755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Sending DeleteTablet for 3 replicas of tablet da4e8eab257845b39bdb98c9c3e02e26
I20260927 21:42:40.038794 15014 tablet_service.cc:1514] Processing DeleteTablet for tablet da4e8eab257845b39bdb98c9c3e02e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:40 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:34864
I20260927 21:42:40.038811 14880 tablet_service.cc:1514] Processing DeleteTablet for tablet da4e8eab257845b39bdb98c9c3e02e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:40 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:34666
I20260927 21:42:40.038902 15148 tablet_service.cc:1514] Processing DeleteTablet for tablet da4e8eab257845b39bdb98c9c3e02e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:40 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:42674
I20260927 21:42:40.040421 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14817
I20260927 21:42:40.050112 15256 tablet_replica.cc:330] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: stopping tablet replica
I20260927 21:42:40.050791 15256 raft_consensus.cc:2229] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:42:40.051177 15254 tablet_replica.cc:330] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: stopping tablet replica
I20260927 21:42:40.051539 15256 raft_consensus.cc:2258] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:40.051810 15254 raft_consensus.cc:2229] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:42:40.052461 15254 raft_consensus.cc:2258] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:40.053558 15255 tablet_replica.cc:330] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: stopping tablet replica
I20260927 21:42:40.054107 15256 ts_tablet_manager.cc:1905] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:40.055213 15254 ts_tablet_manager.cc:1905] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:40.056813 15255 raft_consensus.cc:2229] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:42:40.057648 15255 raft_consensus.cc:2258] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:40.062237 15256 ts_tablet_manager.cc:1918] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:40.062517 15256 log.cc:1196] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/da4e8eab257845b39bdb98c9c3e02e26
I20260927 21:42:40.063195 15256 ts_tablet_manager.cc:1939] T da4e8eab257845b39bdb98c9c3e02e26 P cc889ac6795243d98ea72543575f8f80: Deleting consensus metadata
I20260927 21:42:40.065498 14742 catalog_manager.cc:4841] TS cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717): tablet da4e8eab257845b39bdb98c9c3e02e26 (table test-table [id=296881c1ff444bec9311529381981e5f]) successfully deleted
I20260927 21:42:40.066920 15255 ts_tablet_manager.cc:1905] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:40.067169 15254 ts_tablet_manager.cc:1918] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:40.067445 15254 log.cc:1196] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/da4e8eab257845b39bdb98c9c3e02e26
I20260927 21:42:40.068106 15254 ts_tablet_manager.cc:1939] T da4e8eab257845b39bdb98c9c3e02e26 P da00c0408b0b4856b99c18f4f1d325a2: Deleting consensus metadata
Sep 27 21:42:40 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:42:40.070226 14741 catalog_manager.cc:4841] TS da00c0408b0b4856b99c18f4f1d325a2 (127.0.202.130:41101): tablet da4e8eab257845b39bdb98c9c3e02e26 (table test-table [id=296881c1ff444bec9311529381981e5f]) successfully deleted
I20260927 21:42:40.075047 15255 ts_tablet_manager.cc:1918] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:40.075291 15255 log.cc:1196] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/da4e8eab257845b39bdb98c9c3e02e26
I20260927 21:42:40.075938 15255 ts_tablet_manager.cc:1939] T da4e8eab257845b39bdb98c9c3e02e26 P daba5937ef12466b82364ab43f46d203: Deleting consensus metadata
I20260927 21:42:40.077685 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:40.041916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.0.202.129:41717)
0927 21:42:40.042145 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:40.042155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:40.057666 (+ 15511us) server_negotiation.cc:407] Connection header received
0927 21:42:40.058190 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:40.058196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:40.058293 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:40.058373 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:40.060285 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:40.061408 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:40.063432 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:40.064109 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:40.064208 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:40.070010 (+ 5802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:40.070030 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:40.070048 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:40.070105 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:40.074136 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:40.075086 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:40.075096 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:40.075100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:40.075329 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:40.075812 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:40.075818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:40.075822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:40.076357 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:40.076683 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:40.077030 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 21:42:40.077272 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:42:40.077883 14743 catalog_manager.cc:4841] TS daba5937ef12466b82364ab43f46d203 (127.0.202.131:43879): tablet da4e8eab257845b39bdb98c9c3e02e26 (table test-table [id=296881c1ff444bec9311529381981e5f]) successfully deleted
I20260927 21:42:40.078686 14940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:40.170553 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14817
I20260927 21:42:40.189085 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14951
Sep 27 21:42:40 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:42:40.206698 15227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:40.190128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.0.202.130:41101)
0927 21:42:40.190339 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:40.190345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:40.190367 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:42:40.190562 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:40.190567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:40.190645 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:40.190758 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:40.192025 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:40.192898 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:40.194269 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:40.194675 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:40.194821 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:40.199805 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:40.199823 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:40.199841 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:40.199888 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:40.203615 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:40.204516 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:40.204528 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:40.204532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:40.204679 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:40.205086 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:40.205092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:40.205096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:40.205488 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:40.205809 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:40.206126 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 21:42:40.206355 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:42:40.207518 15074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:40.297832 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14951
I20260927 21:42:40.316169 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 15085
Sep 27 21:42:40 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:42:40.336122 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:40.317146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.202.131:43879)
0927 21:42:40.317378 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:40.317385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:40.317411 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:42:40.317541 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:40.317547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:40.317662 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:40.317785 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:40.319438 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:40.320728 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:40.322329 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:40.323044 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:40.323191 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:40.328585 (+ 5394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:40.328599 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:40.328619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:40.328662 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:40.332507 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:40.333565 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:40.333577 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:40.333581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:40.333814 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:40.334270 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:40.334274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:40.334276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:40.334839 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:40.335209 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:40.335511 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:42:40.335691 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 21:42:40.337029 15208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:40.461462 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 15085
I20260927 21:42:40.481452 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 14724
I20260927 21:42:40.482556 14786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:40.599352 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 14724
I20260927 21:42:40.621148 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35289
--webserver_interface=127.0.202.190
--webserver_port=46433
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:40.984447 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:40.984787 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:40.984949 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:40.993249 15262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:40.993374 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:40.993465 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:40.993556 15262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:40.993659 15262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:41.003988 15262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:35289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=46433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:41.005406 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:41.006803 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:41.030972 15268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:41.030910 15271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:41.030892 15269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:41.031903 15262 server_base.cc:1034] running on GCE node
I20260927 21:42:41.032893 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:41.034616 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:41.035883 15262 hybrid_clock.cc:648] HybridClock initialized: now 1790545361035812 us; error 50 us; skew 500 ppm
I20260927 21:42:41.036366 15262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:41 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:41.040457 15262 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:42:41.042384 15262 webserver.cc:460] Webserver started at http://127.0.202.190:46433/ using document root <none> and password file <none>
I20260927 21:42:41.043294 15262 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:41.043489 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:41.050500 15262 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:42:41.054103 15278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:41.055819 15262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:42:41.056077 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ed8c7d09c0474daa85dd1581b092cf38"
format_stamp: "Formatted at 2026-09-27 21:42:37 on dist-test-slave-25dq"
server_key: "62aca94e136ecbac1e6387f499eeb6c0"
server_key_iv: "3c80a0d6ead12e931b59fcfa0ea095d6"
server_key_version: "encryptionkey@0"
I20260927 21:42:41.057358 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:41.071765 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:41.076444 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:41.076733 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:41.096728 15262 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:35289
I20260927 21:42:41.096755 15329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:35289 every 8 connection(s)
I20260927 21:42:41.098423 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:42:41.100912 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 15262
I20260927 21:42:41.101954 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:41717
--local_ip_for_outbound_sockets=127.0.202.129
--tserver_master_addrs=127.0.202.190:35289
--webserver_port=33819
--webserver_interface=127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:41.109232 15330 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:42:41.113224 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Bootstrap starting.
I20260927 21:42:41.135324 15330 log.cc:824] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:41.148041 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:42:41.148720 15330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Bootstrap complete.
I20260927 21:42:41.158830 15330 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:41.159734 15330 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8c7d09c0474daa85dd1581b092cf38, State: Initialized, Role: FOLLOWER
I20260927 21:42:41.160522 15330 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:41.160804 15330 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:42:41.160945 15330 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:42:41.161126 15330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:42:41.165547 15330 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:41.166090 15330 leader_election.cc:302] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8c7d09c0474daa85dd1581b092cf38; no voters:
I20260927 21:42:41.166756 15330 leader_election.cc:288] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:42:41.166888 15334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:42:41.168033 15334 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 LEADER]: Becoming Leader. State: Replica: ed8c7d09c0474daa85dd1581b092cf38, State: Running, Role: LEADER
I20260927 21:42:41.168704 15334 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:41.169180 15330 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:41.171291 15336 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8c7d09c0474daa85dd1581b092cf38. Latest consensus state: current_term: 2 leader_uuid: "ed8c7d09c0474daa85dd1581b092cf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } } }
I20260927 21:42:41.171378 15335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed8c7d09c0474daa85dd1581b092cf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } } }
I20260927 21:42:41.171684 15336 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:41.171773 15335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:41.181644 15340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:42:41.191018 15340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:42:41.192790 15340 catalog_manager.cc:1259] Loaded cluster ID: c98adfafe612403eb3860afe0ed9b174
I20260927 21:42:41.192986 15340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:42:41.196813 15340 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 21:42:41.215345 15340 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 21:42:41.226565 15340 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:42:41.228398 15340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Loaded TSK: 0
I20260927 21:42:41.230042 15340 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:42:41.472000 15332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:41.472371 15332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:41.472528 15332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:41.480047 15332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:41.480202 15332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:41.480402 15332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:42:41.490607 15332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:41717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=33819
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:41.492038 15332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:41.493465 15332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:41.509140 15357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:41.509171 15358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:41.509431 15360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:41.510249 15332 server_base.cc:1034] running on GCE node
I20260927 21:42:41.511015 15332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:41.512210 15332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:41.513504 15332 hybrid_clock.cc:648] HybridClock initialized: now 1790545361513428 us; error 67 us; skew 500 ppm
I20260927 21:42:41.513931 15332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:41 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:41.524708 15332 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:42:41.526816 15332 webserver.cc:460] Webserver started at http://127.0.202.129:33819/ using document root <none> and password file <none>
I20260927 21:42:41.527630 15332 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:41.527817 15332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:41.533792 15332 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:42:41.536768 15367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:41.538674 15332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:42:41.538887 15332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "cc889ac6795243d98ea72543575f8f80"
format_stamp: "Formatted at 2026-09-27 21:42:38 on dist-test-slave-25dq"
server_key: "0979ce6e1791f58f5799667c129674de"
server_key_iv: "d222c89d34b2894c023952daf905cdb1"
server_key_version: "encryptionkey@0"
I20260927 21:42:41.539991 15332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:41.562830 15332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:41.567437 15332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:41.567728 15332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:41.568832 15332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:41.570309 15332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:41.570429 15332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:41.570561 15332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:41.570645 15332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:41.605531 15332 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41717
I20260927 21:42:41.605620 15479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41717 every 8 connection(s)
I20260927 21:42:41.607049 15332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:42:41.609730 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 15332
I20260927 21:42:41.610494 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:41101
--local_ip_for_outbound_sockets=127.0.202.130
--tserver_master_addrs=127.0.202.190:35289
--webserver_port=43909
--webserver_interface=127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:42:41 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:41.643925 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:41.610472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:56653)
0927 21:42:41.611275 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:41.611314 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:41.613238 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:41.614828 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:41.614842 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:41.615466 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:41.616173 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:41.616201 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:41.618676 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:41.618681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:41.620031 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:41.620044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:41.621210 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:41.621218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:41.621497 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:41.627880 (+ 6383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:41.627918 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:41.631410 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:41.638062 (+ 6652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:41.638073 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:41.638094 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:41.638868 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:41.639487 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:41.639492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:41.639497 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:41.639778 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:41.640941 (+ 1163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:41.640950 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:41.641446 (+ 496us) client_negotiation.cc:769] Sending connection context
0927 21:42:41.641776 (+ 330us) client_negotiation.cc:240] Negotiation successful
0927 21:42:41.642367 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
I20260927 21:42:41.646639 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:41.610060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:56653 (local address 127.0.202.190:35289)
0927 21:42:41.610693 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:41.610705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:41.611487 (+ 782us) server_negotiation.cc:407] Connection header received
0927 21:42:41.613863 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:41.613878 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:41.614244 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:41.614648 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:41.616526 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:41.618505 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:41.620196 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:41.620960 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:41.621162 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:41.631858 (+ 10696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:41.631875 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:41.631902 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:41.631988 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:41.637835 (+ 5847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:41.639056 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:41.639070 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:41.639081 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:41.639321 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:41.639948 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:41.639954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:41.639960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:41.640766 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:41.640978 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:41.642153 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0927 21:42:41.645100 (+ 2947us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
I20260927 21:42:41.650383 15480 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35289
I20260927 21:42:41.650754 15480 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:41.651845 15480 heartbeater.cc:511] Master 127.0.202.190:35289 requested a full tablet report, sending...
I20260927 21:42:41.657022 15295 ts_manager.cc:194] Registered new tserver with Master: cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717)
I20260927 21:42:41.659619 15295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:56653
W20260927 21:42:41.966652 15484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:41.966958 15484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:41.967098 15484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:41.975144 15484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:41.975268 15484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:41.975432 15484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:42:41.985878 15484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:41101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=43909
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:41.987205 15484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:41.988576 15484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:42.002141 15489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:42.002128 15490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:42.002435 15492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:42.003044 15484 server_base.cc:1034] running on GCE node
I20260927 21:42:42.003723 15484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:42.004978 15484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:42.006254 15484 hybrid_clock.cc:648] HybridClock initialized: now 1790545362006156 us; error 81 us; skew 500 ppm
I20260927 21:42:42.006678 15484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:42 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:42.009964 15484 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:42:42.011565 15484 webserver.cc:460] Webserver started at http://127.0.202.130:43909/ using document root <none> and password file <none>
I20260927 21:42:42.012226 15484 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:42.012403 15484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:42.018159 15484 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:42:42.021420 15499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:42.023077 15484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:42:42.023283 15484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "da00c0408b0b4856b99c18f4f1d325a2"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "045f9ae191ba10176576fc5230090e9f"
server_key_iv: "6ea68bdd0855f1ae0c179f3cdb879f05"
server_key_version: "encryptionkey@0"
I20260927 21:42:42.024497 15484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:42.038653 15484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:42.043496 15484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:42.043788 15484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:42.045038 15484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:42.046475 15484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:42.046600 15484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:42.046732 15484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:42.046818 15484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:42.083763 15484 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:41101
I20260927 21:42:42.083802 15611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:41101 every 8 connection(s)
I20260927 21:42:42.085397 15484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:42:42.095367 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 15484
I20260927 21:42:42.096060 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:43879
--local_ip_for_outbound_sockets=127.0.202.131
--tserver_master_addrs=127.0.202.190:35289
--webserver_port=45527
--webserver_interface=127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:42:42 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:42.110595 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.088158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:49051 (local address 127.0.202.190:35289)
0927 21:42:42.088475 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.088481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.089585 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 21:42:42.091600 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.091606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.091708 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.091828 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:42.093573 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.094350 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.096517 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.097320 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.097421 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.102549 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:42.102557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:42.102563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:42.102609 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:42.106553 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.107628 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.107634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.107639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.107807 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.108399 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.108408 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.108413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.108847 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:42.108999 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.109938 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.110208 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:42:42.111944 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.088712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:49051)
0927 21:42:42.089420 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:42.089466 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:42.091305 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:42.092096 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:42.092105 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:42.092757 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:42.093390 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.093408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.094509 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.094521 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.096323 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.096340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.097767 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.097775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.098096 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.098943 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:42.098975 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:42.102261 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:42.106737 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:42.106751 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:42.106771 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:42.107475 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:42.107939 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:42.107945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:42.107949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:42.108217 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:42.108992 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:42.109004 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:42.109499 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 21:42:42.109802 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 21:42:42.110400 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":272,"threads_started":1}
I20260927 21:42:42.113687 15612 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35289
I20260927 21:42:42.114123 15612 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:42.115135 15612 heartbeater.cc:511] Master 127.0.202.190:35289 requested a full tablet report, sending...
I20260927 21:42:42.117091 15295 ts_manager.cc:194] Registered new tserver with Master: da00c0408b0b4856b99c18f4f1d325a2 (127.0.202.130:41101)
I20260927 21:42:42.118585 15295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:49051
W20260927 21:42:42.447466 15615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:42.447808 15615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:42.447983 15615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:42.455451 15615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:42.455611 15615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:42.455792 15615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:42:42.466655 15615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:43879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=45527
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:42.468076 15615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:42.469565 15615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:42.484215 15620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:42.484177 15623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:42.484246 15621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:42.484933 15615 server_base.cc:1034] running on GCE node
I20260927 21:42:42.485610 15615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:42.486694 15615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:42.487957 15615 hybrid_clock.cc:648] HybridClock initialized: now 1790545362487875 us; error 60 us; skew 500 ppm
I20260927 21:42:42.488384 15615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:42 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:42.491987 15615 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:42:42.493857 15615 webserver.cc:460] Webserver started at http://127.0.202.131:45527/ using document root <none> and password file <none>
I20260927 21:42:42.494611 15615 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:42.494772 15615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:42.500566 15615 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:42:42.503871 15630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:42.505752 15615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:42:42.506042 15615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "daba5937ef12466b82364ab43f46d203"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "520941fd3085c899c990d9f88364e0d4"
server_key_iv: "28dcba70c7b979d2b26bc143e0a304c7"
server_key_version: "encryptionkey@0"
I20260927 21:42:42.507164 15615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:42.537473 15615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:42.542511 15615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:42.542829 15615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:42.543951 15615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:42.545797 15615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:42.545929 15615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:42.546075 15615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:42.546178 15615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:42.583724 15615 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:43879
I20260927 21:42:42.583819 15742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:43879 every 8 connection(s)
I20260927 21:42:42.585353 15615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:42:42.593451 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 15615
Sep 27 21:42:42 dist-test-slave-25dq krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:42:42.611585 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.588171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:47501 (local address 127.0.202.190:35289)
0927 21:42:42.588431 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.588439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.589789 (+ 1350us) server_negotiation.cc:407] Connection header received
0927 21:42:42.591807 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.591812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.591886 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.591983 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:42.593727 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.594699 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.597513 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.598306 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.598452 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.603502 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:42.603507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:42.603510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:42.603543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:42.606864 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.607911 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.607917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.607922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.608077 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.608705 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.608712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.608718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.609103 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:42.609256 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.611003 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.611241 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":83}
I20260927 21:42:42.613018 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.588807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:47501)
0927 21:42:42.589615 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:42.589659 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:42.591487 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:42.592212 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:42.592221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:42.592889 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:42:42.593501 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.593546 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.594931 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.594936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.597299 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.597316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.598570 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.598576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.598796 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.599707 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:42:42.599741 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:42:42.603163 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:42:42.607035 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:42.607046 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:42.607065 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:42.607755 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:42.608207 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:42:42.608213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:42:42.608224 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:42:42.608549 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:42:42.610071 (+ 1522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:42:42.610080 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:42:42.610529 (+ 449us) client_negotiation.cc:769] Sending connection context
0927 21:42:42.610871 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 21:42:42.611489 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
I20260927 21:42:42.613193 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.594437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.0.202.190:35289)
0927 21:42:42.595202 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.595206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.595223 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:42:42.595266 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.595269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.595327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.595414 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:42.597523 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.598812 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.601186 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.601989 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.602148 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.605766 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:42.605773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:42.605778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:42.605829 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:42.609124 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.610038 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.610042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.610044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.610161 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.610597 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.610603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.610606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.610926 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:42.611243 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.612488 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.612764 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":677,"thread_start_us":543,"threads_started":1}
I20260927 21:42:42.614552 15743 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35289
I20260927 21:42:42.614914 15743 heartbeater.cc:464] Registering TS with master...
I20260927 21:42:42.615681 15743 heartbeater.cc:511] Master 127.0.202.190:35289 requested a full tablet report, sending...
I20260927 21:42:42.617251 15294 ts_manager.cc:194] Registered new tserver with Master: daba5937ef12466b82364ab43f46d203 (127.0.202.131:43879)
I20260927 21:42:42.618206 15294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:47501
I20260927 21:42:42.629036 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:42:42.639411 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.630181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.0.202.190:35289)
0927 21:42:42.630521 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.630534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.631081 (+ 547us) server_negotiation.cc:407] Connection header received
0927 21:42:42.631362 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.631368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.631610 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.631836 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:42:42.632447 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.633754 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.636167 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.637011 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.637172 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.637903 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:42:42.638659 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:42:42.638779 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.638936 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.639049 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:42:42.640545 15294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:42:42.648270 15294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:42:42.663056 15480 heartbeater.cc:503] Master 127.0.202.190:35289 was elected leader, sending a full tablet report...
I20260927 21:42:42.670622 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.660761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.0.202.131:43879)
0927 21:42:42.663844 (+ 3083us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.663852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.663884 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:42:42.663980 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.663988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.664256 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.665191 (+ 935us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:42.665398 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.666399 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.668314 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.669349 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.669500 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.669744 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.670075 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.670240 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2911,"thread_start_us":3899,"threads_started":1}
I20260927 21:42:42.670624 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.660495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:43879 (local address 127.0.0.1:42690)
0927 21:42:42.662373 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:42.662446 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:42.662678 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:42.664527 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:42.664534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:42.664575 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:42.665020 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.665031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.666550 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.666558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.668124 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.668137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.669566 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.669572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.669778 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.669840 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 21:42:42.670034 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 21:42:42.670228 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"thread_start_us":658,"threads_started":1}
I20260927 21:42:42.675340 15678 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9fbfcbc2d14c628c50c5d7d156f051 (DEFAULT_TABLE table=test-table [id=f34c22a3f5bf462bbb7c33c769180dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:42.677138 15678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9fbfcbc2d14c628c50c5d7d156f051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:42.678956 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.661382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41717 (local address 127.0.0.1:34684)
0927 21:42:42.671789 (+ 10407us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:42.671806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:42.671940 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:42.672562 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:42.672567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:42.672592 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:42.672975 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.672984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.674508 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.674513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.676097 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.676108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.678173 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.678177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.678290 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.678311 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:42:42.678419 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:42:42.678560 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10293,"thread_start_us":10157,"threads_started":1}
I20260927 21:42:42.679054 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.661802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.0.202.129:41717)
0927 21:42:42.663387 (+ 1585us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.663396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.671903 (+ 8507us) server_negotiation.cc:407] Connection header received
0927 21:42:42.672094 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.672104 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.672369 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.672563 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:42.673127 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.674366 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.676282 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.677701 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.677789 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.678008 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.678509 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.678737 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1415,"thread_start_us":1188,"threads_started":1}
I20260927 21:42:42.681821 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.660742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:41101 (local address 127.0.0.1:34890)
0927 21:42:42.671185 (+ 10443us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:42.671205 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:42.671309 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:42.673476 (+ 2167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:42.673484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:42.673511 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:42.673662 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.673671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.675622 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.675631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.677693 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.677708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.680960 (+ 3252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.680967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.681193 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.681237 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:42:42.681346 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:42:42.681509 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10347,"thread_start_us":11495,"threads_started":1}
I20260927 21:42:42.682132 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.661877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.0.202.130:41101)
0927 21:42:42.671573 (+ 9696us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.671582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.671615 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:42:42.671722 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.671732 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.672061 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.673476 (+ 1415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:42.673809 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.675432 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.677999 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.680732 (+ 2733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.680939 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.681161 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.681493 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.681713 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9476,"thread_start_us":8756,"threads_started":1}
I20260927 21:42:42.683934 15415 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9fbfcbc2d14c628c50c5d7d156f051 (DEFAULT_TABLE table=test-table [id=f34c22a3f5bf462bbb7c33c769180dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:42.685537 15415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9fbfcbc2d14c628c50c5d7d156f051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:42.686236 15547 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9fbfcbc2d14c628c50c5d7d156f051 (DEFAULT_TABLE table=test-table [id=f34c22a3f5bf462bbb7c33c769180dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:42:42.687640 15547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9fbfcbc2d14c628c50c5d7d156f051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:42:42.694816 15759 tablet_bootstrap.cc:492] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Bootstrap starting.
I20260927 21:42:42.699689 15759 tablet_bootstrap.cc:654] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:42.700088 15760 tablet_bootstrap.cc:492] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Bootstrap starting.
I20260927 21:42:42.701278 15759 log.cc:824] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:42.704792 15759 tablet_bootstrap.cc:492] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: No bootstrap required, opened a new log
I20260927 21:42:42.704828 15760 tablet_bootstrap.cc:654] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:42.705122 15759 ts_tablet_manager.cc:1397] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260927 21:42:42.706669 15760 log.cc:824] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:42.710263 15760 tablet_bootstrap.cc:492] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: No bootstrap required, opened a new log
I20260927 21:42:42.710598 15760 ts_tablet_manager.cc:1397] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 21:42:42.712411 15762 tablet_bootstrap.cc:492] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Bootstrap starting.
I20260927 21:42:42.714974 15759 raft_consensus.cc:348] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.715456 15759 raft_consensus.cc:374] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:42.715564 15759 raft_consensus.cc:729] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daba5937ef12466b82364ab43f46d203, State: Initialized, Role: FOLLOWER
I20260927 21:42:42.716336 15759 consensus_queue.cc:260] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.716833 15762 tablet_bootstrap.cc:654] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Neither blocks nor log segments found. Creating new log.
I20260927 21:42:42.717730 15743 heartbeater.cc:503] Master 127.0.202.190:35289 was elected leader, sending a full tablet report...
I20260927 21:42:42.718261 15762 log.cc:824] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:42.718439 15759 ts_tablet_manager.cc:1428] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260927 21:42:42.721937 15762 tablet_bootstrap.cc:492] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: No bootstrap required, opened a new log
I20260927 21:42:42.722270 15762 ts_tablet_manager.cc:1397] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260927 21:42:42.722745 15760 raft_consensus.cc:348] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.723275 15760 raft_consensus.cc:374] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:42.723413 15760 raft_consensus.cc:729] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc889ac6795243d98ea72543575f8f80, State: Initialized, Role: FOLLOWER
I20260927 21:42:42.724129 15760 consensus_queue.cc:260] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.729270 15760 ts_tablet_manager.cc:1428] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260927 21:42:42.730804 15762 raft_consensus.cc:348] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.731271 15762 raft_consensus.cc:374] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:42:42.731400 15762 raft_consensus.cc:729] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da00c0408b0b4856b99c18f4f1d325a2, State: Initialized, Role: FOLLOWER
I20260927 21:42:42.731945 15762 consensus_queue.cc:260] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.733153 15612 heartbeater.cc:503] Master 127.0.202.190:35289 was elected leader, sending a full tablet report...
I20260927 21:42:42.733759 15762 ts_tablet_manager.cc:1428] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260927 21:42:42.837129 15744 tablet.cc:2369] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:42:42.837357 15613 tablet.cc:2369] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:42:42.859449 15481 tablet.cc:2369] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:42:42.872498 15767 raft_consensus.cc:482] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:42:42.872830 15767 raft_consensus.cc:504] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.874603 15767 leader_election.cc:288] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daba5937ef12466b82364ab43f46d203 (127.0.202.131:43879), cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717)
I20260927 21:42:42.884745 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.874986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:56113 (local address 127.0.202.131:43879)
0927 21:42:42.875196 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.875203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.875904 (+ 701us) server_negotiation.cc:407] Connection header received
0927 21:42:42.876132 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.876138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.876217 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.876355 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:42.877123 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.878309 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.881852 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.883153 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.883318 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.883396 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.884255 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.884439 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:42:42.884770 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.875477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:42267 (local address 127.0.202.129:41717)
0927 21:42:42.875703 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.875710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.876052 (+ 342us) server_negotiation.cc:407] Connection header received
0927 21:42:42.876191 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.876197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.876275 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.876376 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:42:42.877217 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.878345 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.881849 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.883134 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.883318 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.883396 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.884255 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.884516 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:42:42.885094 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.874905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:43879 (local address 127.0.202.130:56113)
0927 21:42:42.875766 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:42.875793 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:42.875964 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:42.876385 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:42.876389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:42.876419 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:42.876950 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.876968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.878536 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.878540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.881641 (+ 3101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.881662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.883570 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.883578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.883954 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.884077 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 21:42:42.884191 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:42:42.884433 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":447,"thread_start_us":382,"threads_started":1}
I20260927 21:42:42.885116 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.875060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41717 (local address 127.0.202.130:42267)
0927 21:42:42.875922 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:42.875938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:42:42.876073 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:42:42.876392 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:42:42.876395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:42:42.876447 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:42:42.877066 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.877076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.878477 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.878483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.881641 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:42:42.881662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.883572 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.883578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:42:42.883950 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.884074 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 21:42:42.884191 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 21:42:42.884433 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":370,"spinlock_wait_cycles":39168,"thread_start_us":561,"threads_started":1}
I20260927 21:42:42.886413 15698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9fbfcbc2d14c628c50c5d7d156f051" candidate_uuid: "da00c0408b0b4856b99c18f4f1d325a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daba5937ef12466b82364ab43f46d203" is_pre_election: true
I20260927 21:42:42.886466 15435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9fbfcbc2d14c628c50c5d7d156f051" candidate_uuid: "da00c0408b0b4856b99c18f4f1d325a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc889ac6795243d98ea72543575f8f80" is_pre_election: true
I20260927 21:42:42.887040 15698 raft_consensus.cc:2454] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da00c0408b0b4856b99c18f4f1d325a2 in term 0.
I20260927 21:42:42.887151 15435 raft_consensus.cc:2454] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da00c0408b0b4856b99c18f4f1d325a2 in term 0.
I20260927 21:42:42.887794 15503 leader_election.cc:302] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cc889ac6795243d98ea72543575f8f80, da00c0408b0b4856b99c18f4f1d325a2, daba5937ef12466b82364ab43f46d203; no voters:
I20260927 21:42:42.888213 15767 raft_consensus.cc:2781] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:42:42.888481 15767 raft_consensus.cc:482] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:42:42.888581 15767 raft_consensus.cc:3037] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:42.890928 15767 raft_consensus.cc:504] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.891587 15767 leader_election.cc:288] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [CANDIDATE]: Term 1 election: Requested vote from peers daba5937ef12466b82364ab43f46d203 (127.0.202.131:43879), cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717)
I20260927 21:42:42.892129 15435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9fbfcbc2d14c628c50c5d7d156f051" candidate_uuid: "da00c0408b0b4856b99c18f4f1d325a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc889ac6795243d98ea72543575f8f80"
I20260927 21:42:42.892129 15698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9fbfcbc2d14c628c50c5d7d156f051" candidate_uuid: "da00c0408b0b4856b99c18f4f1d325a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daba5937ef12466b82364ab43f46d203"
I20260927 21:42:42.892390 15435 raft_consensus.cc:3037] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:42.892395 15698 raft_consensus.cc:3037] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:42:42.896034 15435 raft_consensus.cc:2454] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da00c0408b0b4856b99c18f4f1d325a2 in term 1.
I20260927 21:42:42.896034 15698 raft_consensus.cc:2454] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da00c0408b0b4856b99c18f4f1d325a2 in term 1.
I20260927 21:42:42.896687 15502 leader_election.cc:302] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cc889ac6795243d98ea72543575f8f80, da00c0408b0b4856b99c18f4f1d325a2, daba5937ef12466b82364ab43f46d203; no voters:
I20260927 21:42:42.897128 15767 raft_consensus.cc:2781] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:42:42.897712 15767 raft_consensus.cc:686] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 LEADER]: Becoming Leader. State: Replica: da00c0408b0b4856b99c18f4f1d325a2, State: Running, Role: LEADER
I20260927 21:42:42.898504 15767 consensus_queue.cc:237] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } }
I20260927 21:42:42.903661 15295 catalog_manager.cc:5495] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to da00c0408b0b4856b99c18f4f1d325a2 (127.0.202.130). New cstate: current_term: 1 leader_uuid: "da00c0408b0b4856b99c18f4f1d325a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da00c0408b0b4856b99c18f4f1d325a2" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 } health_report { overall_health: UNKNOWN } } }
I20260927 21:42:42.925040 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.919584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.0.202.130:41101)
0927 21:42:42.919759 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.919763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.919780 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:42:42.919966 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.919971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.920051 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.920179 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:42:42.920570 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.921287 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.922917 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.923303 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.923400 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.923918 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:42:42.924524 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:42:42.924647 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.924765 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.924833 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:42:42.933295 15435 raft_consensus.cc:1261] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Refusing update from remote peer da00c0408b0b4856b99c18f4f1d325a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:42.933296 15698 raft_consensus.cc:1261] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 1 FOLLOWER]: Refusing update from remote peer da00c0408b0b4856b99c18f4f1d325a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:42:42.934296 15770 consensus_queue.cc:1035] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc889ac6795243d98ea72543575f8f80" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:42.934628 15767 consensus_queue.cc:1035] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daba5937ef12466b82364ab43f46d203" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:42:42.947944 15775 mvcc.cc:205] Tried to move back new op lower bound from 7334073806564900864 to 7334073806436409344. Current Snapshot: MvccSnapshot[applied={T|T < 7334073806564900864}]
I20260927 21:42:42.960683 15295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:42:42.960987 15295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:42:42.964735 15295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Sending DeleteTablet for 3 replicas of tablet 7e9fbfcbc2d14c628c50c5d7d156f051
I20260927 21:42:42.965909 15678 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9fbfcbc2d14c628c50c5d7d156f051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:42 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:42690
I20260927 21:42:42.966005 15415 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9fbfcbc2d14c628c50c5d7d156f051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:42 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:34684
I20260927 21:42:42.966768 15547 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9fbfcbc2d14c628c50c5d7d156f051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:42:42 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:34890
I20260927 21:42:42.966900 15784 tablet_replica.cc:330] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: stopping tablet replica
I20260927 21:42:42.967481 15784 raft_consensus.cc:2229] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:42:42.968078 15784 raft_consensus.cc:2258] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:42.968969 15783 tablet_replica.cc:330] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: stopping tablet replica
I20260927 21:42:42.969404 15783 raft_consensus.cc:2229] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:42:42.969939 15783 raft_consensus.cc:2258] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:42.970819 15784 ts_tablet_manager.cc:1905] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:42.970970 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 15332
I20260927 21:42:42.974858 15783 ts_tablet_manager.cc:1905] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:42.978400 15784 ts_tablet_manager.cc:1918] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:42.978595 15784 log.cc:1196] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/7e9fbfcbc2d14c628c50c5d7d156f051
I20260927 21:42:42.978916 15785 tablet_replica.cc:330] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: stopping tablet replica
I20260927 21:42:42.979315 15784 ts_tablet_manager.cc:1939] T 7e9fbfcbc2d14c628c50c5d7d156f051 P cc889ac6795243d98ea72543575f8f80: Deleting consensus metadata
I20260927 21:42:42.979385 15785 raft_consensus.cc:2229] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:42:42.980085 15785 raft_consensus.cc:2258] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:42:42.981659 15281 catalog_manager.cc:4841] TS cc889ac6795243d98ea72543575f8f80 (127.0.202.129:41717): tablet 7e9fbfcbc2d14c628c50c5d7d156f051 (table test-table [id=f34c22a3f5bf462bbb7c33c769180dcf]) successfully deleted
I20260927 21:42:42.982421 15785 ts_tablet_manager.cc:1905] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:42:42.986121 15783 ts_tablet_manager.cc:1918] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:42.986310 15783 log.cc:1196] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/7e9fbfcbc2d14c628c50c5d7d156f051
I20260927 21:42:42.986811 15783 ts_tablet_manager.cc:1939] T 7e9fbfcbc2d14c628c50c5d7d156f051 P daba5937ef12466b82364ab43f46d203: Deleting consensus metadata
I20260927 21:42:42.988713 15282 catalog_manager.cc:4841] TS daba5937ef12466b82364ab43f46d203 (127.0.202.131:43879): tablet 7e9fbfcbc2d14c628c50c5d7d156f051 (table test-table [id=f34c22a3f5bf462bbb7c33c769180dcf]) successfully deleted
I20260927 21:42:42.989898 15785 ts_tablet_manager.cc:1918] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:42:42.990090 15785 log.cc:1196] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/7e9fbfcbc2d14c628c50c5d7d156f051
I20260927 21:42:42.990406 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:42.971883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.202.129:41717)
0927 21:42:42.972062 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:42.972068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:42.972381 (+ 313us) server_negotiation.cc:407] Connection header received
0927 21:42:42.972615 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:42.972627 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:42.972759 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:42.972893 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:42.974651 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.976684 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.978661 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:42.979311 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:42.979657 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:42.983099 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:42.983116 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:42.983135 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:42.983181 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:42.987159 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.988050 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.988060 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.988066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.988228 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:42.988709 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:42.988714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:42.988717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:42.989158 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:42.989451 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:42.989741 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 21:42:42.990021 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:42:42.990643 15785 ts_tablet_manager.cc:1939] T 7e9fbfcbc2d14c628c50c5d7d156f051 P da00c0408b0b4856b99c18f4f1d325a2: Deleting consensus metadata
I20260927 21:42:42.991174 15475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:42.992511 15280 catalog_manager.cc:4841] TS da00c0408b0b4856b99c18f4f1d325a2 (127.0.202.130:41101): tablet 7e9fbfcbc2d14c628c50c5d7d156f051 (table test-table [id=f34c22a3f5bf462bbb7c33c769180dcf]) successfully deleted
I20260927 21:42:43.085175 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 15332
I20260927 21:42:43.112771 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 15484
I20260927 21:42:43.129034 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:43.113632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.0.202.130:41101)
0927 21:42:43.113843 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:43.113850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:43.114091 (+ 241us) server_negotiation.cc:407] Connection header received
0927 21:42:43.114159 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:43.114165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:43.114247 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:43.114346 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:43.116229 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:43.117535 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:43.119116 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:43.119559 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:43.119705 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:43.122623 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:43.122642 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:43.122664 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:43.122698 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:43.125828 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:43.126774 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:43.126786 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:43.126788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:43.126930 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:43.127339 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:43.127342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:43.127344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:43.127732 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:43.128003 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:43.128427 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 21:42:43.128666 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:42:43.130036 15607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:43.248112 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 15484
I20260927 21:42:43.269440 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 15615
I20260927 21:42:43.285035 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:43.270247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.202.131:43879)
0927 21:42:43.270500 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 21:42:43.270510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:42:43.270602 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:42:43.270844 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:42:43.270849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:42:43.270919 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:42:43.271056 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:42:43.272654 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:43.273435 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:43.274897 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:42:43.275329 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:42:43.275498 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:42:43.278566 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:42:43.278583 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:42:43.278601 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:42:43.278639 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:42:43.281825 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:43.282744 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:43.282752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:43.282754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:43.282930 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:42:43.283394 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:42:43.283400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:42:43.283404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:42:43.283798 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:42:43.284088 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 21:42:43.284450 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 21:42:43.284633 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 21:42:43.285742 15738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:43.379371 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 15615
I20260927 21:42:43.396214 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 15262
I20260927 21:42:43.397411 15325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:42:43.512578 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 15262
I20260927 21:42:43.534771 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35289
--webserver_interface=127.0.202.190
--webserver_port=46433
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:42:43.880810 15790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:43.881129 15790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:43.881305 15790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:43.889473 15790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:42:43.889590 15790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:43.889665 15790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:43.889750 15790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:42:43.889809 15790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:42:43.899945 15790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:35289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=46433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:43.901441 15790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:43.902837 15790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:43.916226 15795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:43.916170 15796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:43.916249 15798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:43.917202 15790 server_base.cc:1034] running on GCE node
I20260927 21:42:43.918054 15790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:43.919538 15790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:43.920744 15790 hybrid_clock.cc:648] HybridClock initialized: now 1790545363920692 us; error 41 us; skew 500 ppm
I20260927 21:42:43.921077 15790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:43 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:43.924155 15790 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:42:43.925802 15790 webserver.cc:460] Webserver started at http://127.0.202.190:46433/ using document root <none> and password file <none>
I20260927 21:42:43.926528 15790 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:43.926671 15790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:43.932219 15790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:42:43.935186 15805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:43.936587 15790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:42:43.936761 15790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ed8c7d09c0474daa85dd1581b092cf38"
format_stamp: "Formatted at 2026-09-27 21:42:37 on dist-test-slave-25dq"
server_key: "62aca94e136ecbac1e6387f499eeb6c0"
server_key_iv: "3c80a0d6ead12e931b59fcfa0ea095d6"
server_key_version: "encryptionkey@0"
I20260927 21:42:43.937767 15790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:43.950388 15790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:43.954505 15790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:43.954787 15790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:43.973050 15790 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:35289
I20260927 21:42:43.973058 15856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:35289 every 8 connection(s)
I20260927 21:42:43.974598 15790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:42:43.978175 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 15790
I20260927 21:42:43.979039 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:41717
--local_ip_for_outbound_sockets=127.0.202.129
--tserver_master_addrs=127.0.202.190:35289
--webserver_port=33819
--webserver_interface=127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:43.981997 15857 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:42:43.985359 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Bootstrap starting.
I20260927 21:42:44.016880 15857 log.cc:824] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Log is configured to *not* fsync() on all Append() calls
I20260927 21:42:44.039945 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:42:44.040764 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38: Bootstrap complete.
I20260927 21:42:44.052134 15857 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:44.053200 15857 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8c7d09c0474daa85dd1581b092cf38, State: Initialized, Role: FOLLOWER
I20260927 21:42:44.054006 15857 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:44.054260 15857 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:42:44.054382 15857 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:42:44.054582 15857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:42:44.058789 15857 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:44.059327 15857 leader_election.cc:302] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8c7d09c0474daa85dd1581b092cf38; no voters:
I20260927 21:42:44.060024 15857 leader_election.cc:288] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:42:44.060173 15861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:42:44.061445 15861 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [term 3 LEADER]: Becoming Leader. State: Replica: ed8c7d09c0474daa85dd1581b092cf38, State: Running, Role: LEADER
I20260927 21:42:44.062034 15861 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } }
I20260927 21:42:44.062512 15857 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:42:44.064864 15862 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ed8c7d09c0474daa85dd1581b092cf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } } }
I20260927 21:42:44.065356 15862 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:44.067718 15863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8c7d09c0474daa85dd1581b092cf38. Latest consensus state: current_term: 3 leader_uuid: "ed8c7d09c0474daa85dd1581b092cf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8c7d09c0474daa85dd1581b092cf38" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35289 } } }
I20260927 21:42:44.068043 15863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8c7d09c0474daa85dd1581b092cf38 [sys.catalog]: This master's current role is: LEADER
I20260927 21:42:44.069952 15867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:42:44.081893 15867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:42:44.084249 15867 catalog_manager.cc:1259] Loaded cluster ID: c98adfafe612403eb3860afe0ed9b174
I20260927 21:42:44.084429 15867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 21:42:44.087381 15867 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5d313769ad google::LogMessage::Fail() at ??:0
@ 0x7f5d3137988a google::LogMessage::SendToLog() at ??:0
@ 0x7f5d313764de google::LogMessage::Flush() at ??:0
@ 0x7f5d31377a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5d46da9512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260927 21:42:44.107106 15879 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f5d46da6976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5d46e2e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5d46e2e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5d33fc2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5d33fc591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5d33fc5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5d33fa0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5d3727a6db start_thread at ??:0
@ 0x7f5d3021061f clone at ??:0
W20260927 21:42:44.411751 15859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:44.412086 15859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:44.412261 15859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:44.420827 15859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:44.420964 15859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:44.421164 15859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:42:44.432480 15859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:41717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=33819
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:44.433884 15859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:44.435264 15859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:44.449047 15898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:44.449093 15899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:44.449123 15901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:44.450014 15859 server_base.cc:1034] running on GCE node
I20260927 21:42:44.450719 15859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:44.451717 15859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:44.452973 15859 hybrid_clock.cc:648] HybridClock initialized: now 1790545364452930 us; error 63 us; skew 500 ppm
I20260927 21:42:44.453328 15859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:44 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:44.456715 15859 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:42:44.458137 15859 webserver.cc:460] Webserver started at http://127.0.202.129:33819/ using document root <none> and password file <none>
I20260927 21:42:44.458813 15859 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:44.458914 15859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:44.464946 15859 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 21:42:44.467787 15908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:44.469254 15859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:42:44.469398 15859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "cc889ac6795243d98ea72543575f8f80"
format_stamp: "Formatted at 2026-09-27 21:42:38 on dist-test-slave-25dq"
server_key: "0979ce6e1791f58f5799667c129674de"
server_key_iv: "d222c89d34b2894c023952daf905cdb1"
server_key_version: "encryptionkey@0"
I20260927 21:42:44.470405 15859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:44.490283 15859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:44.495082 15859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:44.495330 15859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:44.496402 15859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:44.497895 15859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:44.498015 15859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:44.498096 15859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:44.498152 15859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:44.532725 15859 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41717
I20260927 21:42:44.532778 16020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41717 every 8 connection(s)
I20260927 21:42:44.534184 15859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:42:44.538589 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 15859
I20260927 21:42:44.539294 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:41101
--local_ip_for_outbound_sockets=127.0.202.130
--tserver_master_addrs=127.0.202.190:35289
--webserver_port=43909
--webserver_interface=127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:44.539461 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:44.537443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:46817)
0927 21:42:44.538240 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:44.538466 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
W20260927 21:42:44.540138 16021 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:35289 (0 consecutive failures): Network error: Failed to ping master at 127.0.202.190:35289: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
I20260927 21:42:44.541158 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:44.540786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:35353)
0927 21:42:44.540959 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:44.541047 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 21:42:44.542284 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:44.541974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:57463)
0927 21:42:44.542103 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:44.542175 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:42:44.542578 16021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:42:44.876652 16024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:44.876935 16024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:44.877031 16024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:44.884366 16024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:44.884476 16024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:44.884595 16024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:42:44.895095 16024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:41101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=43909
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:44.896474 16024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:44.897835 16024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:44.911192 16032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:44.911345 16029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:44.911369 16030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:44.912324 16024 server_base.cc:1034] running on GCE node
I20260927 21:42:44.912941 16024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:44.913895 16024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:44.915094 16024 hybrid_clock.cc:648] HybridClock initialized: now 1790545364915045 us; error 37 us; skew 500 ppm
I20260927 21:42:44.915391 16024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:44 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:44.918628 16024 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:42:44.920066 16024 webserver.cc:460] Webserver started at http://127.0.202.130:43909/ using document root <none> and password file <none>
I20260927 21:42:44.920806 16024 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:44.920909 16024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:44.926065 16024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:42:44.928768 16039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:44.930096 16024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:42:44.930226 16024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "da00c0408b0b4856b99c18f4f1d325a2"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "045f9ae191ba10176576fc5230090e9f"
server_key_iv: "6ea68bdd0855f1ae0c179f3cdb879f05"
server_key_version: "encryptionkey@0"
I20260927 21:42:44.931164 16024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:44.960232 16024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:44.964816 16024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:44.965062 16024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:44.966034 16024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:44.967510 16024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:44.967598 16024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:44.967679 16024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:44.967734 16024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:45.002439 16024 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:41101
I20260927 21:42:45.002518 16151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:41101 every 8 connection(s)
I20260927 21:42:45.003906 16024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:42:45.005688 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16024
I20260927 21:42:45.006292 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:43879
--local_ip_for_outbound_sockets=127.0.202.131
--tserver_master_addrs=127.0.202.190:35289
--webserver_port=45527
--webserver_interface=127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:42:45.009258 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:45.007119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:50283)
0927 21:42:45.007972 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:45.008280 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":351,"threads_started":1}
W20260927 21:42:45.009917 16152 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:35289 (0 consecutive failures): Network error: Failed to ping master at 127.0.202.190:35289: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
I20260927 21:42:45.010835 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:45.010528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:35027)
0927 21:42:45.010671 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:45.010737 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:42:45.011763 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:45.011418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:47451)
0927 21:42:45.011537 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:45.011603 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 21:42:45.012006 16152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:42:45.342566 16155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:42:45.342864 16155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:42:45.343030 16155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:42:45.350389 16155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:42:45.350507 16155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:42:45.350685 16155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:42:45.360919 16155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:43879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=45527
--tserver_master_addrs=127.0.202.190:35289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:42:45.362394 16155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:42:45.363852 16155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:42:45.377786 16160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:45.377979 16163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:42:45.377804 16161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:42:45.378727 16155 server_base.cc:1034] running on GCE node
I20260927 21:42:45.379367 16155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:42:45.380513 16155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:42:45.381722 16155 hybrid_clock.cc:648] HybridClock initialized: now 1790545365381666 us; error 44 us; skew 500 ppm
I20260927 21:42:45.382079 16155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:42:45 dist-test-slave-25dq krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:42:45.385318 16155 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:42:45.386888 16155 webserver.cc:460] Webserver started at http://127.0.202.131:45527/ using document root <none> and password file <none>
I20260927 21:42:45.387619 16155 fs_manager.cc:359] Metadata directory not provided
I20260927 21:42:45.387751 16155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:42:45.393985 16155 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:42:45.397837 16170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:45.399318 16155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:42:45.399495 16155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "daba5937ef12466b82364ab43f46d203"
format_stamp: "Formatted at 2026-09-27 21:42:39 on dist-test-slave-25dq"
server_key: "520941fd3085c899c990d9f88364e0d4"
server_key_iv: "28dcba70c7b979d2b26bc143e0a304c7"
server_key_version: "encryptionkey@0"
I20260927 21:42:45.400620 16155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:42:45.425956 16155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:42:45.430665 16155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:42:45.430964 16155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:42:45.431972 16155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:42:45.433367 16155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:42:45.433497 16155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:45.433629 16155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:42:45.433702 16155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:42:45.466833 16155 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:43879
I20260927 21:42:45.466886 16282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:43879 every 8 connection(s)
I20260927 21:42:45.468392 16155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:42:45.470911 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16155
I20260927 21:42:45.473876 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:45.471797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:55287)
0927 21:42:45.472713 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:45.472979 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":400,"threads_started":1}
W20260927 21:42:45.474620 16283 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:35289 (0 consecutive failures): Network error: Failed to ping master at 127.0.202.190:35289: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
I20260927 21:42:45.475474 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:45.475136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:47959)
0927 21:42:45.475302 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:45.475370 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 21:42:45.476486 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:45.476142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:59061)
0927 21:42:45.476260 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:45.476382 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 21:42:45.476780 16283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:42:45.544121 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:45.543151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:40165)
0927 21:42:45.543728 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:45.543871 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":328,"threads_started":1}
I20260927 21:42:46.013588 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:46.012696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:58719)
0927 21:42:46.013200 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:46.013297 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
I20260927 21:42:46.478349 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:46.477508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:42635)
0927 21:42:46.478022 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:46.478118 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":230,"threads_started":1}
I20260927 21:42:46.546043 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:46.545161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:56575)
0927 21:42:46.545723 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:46.545829 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":261,"threads_started":1}
I20260927 21:42:47.015360 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:47.014604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:40209)
0927 21:42:47.015100 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:47.015190 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260927 21:42:47.480229 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:47.479362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:49507)
0927 21:42:47.479816 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:47.479994 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260927 21:42:47.547857 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:47.547053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:58775)
0927 21:42:47.547547 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:47.547643 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
I20260927 21:42:48.017282 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:48.016513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:40001)
0927 21:42:48.016980 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:48.017089 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260927 21:42:48.482031 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:48.481253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:34173)
0927 21:42:48.481730 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:48.481838 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260927 21:42:48.549777 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:48.548907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:33037)
0927 21:42:48.549461 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:48.549558 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":288,"threads_started":1}
I20260927 21:42:49.019129 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:49.018302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:53139)
0927 21:42:49.018825 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:49.018922 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
I20260927 21:42:49.483866 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:49.483080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:59917)
0927 21:42:49.483598 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:49.483687 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260927 21:42:49.551787 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:49.550909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:60035)
0927 21:42:49.551477 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:49.551588 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":295,"threads_started":1}
I20260927 21:42:50.021138 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:50.020231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:42875)
0927 21:42:50.020873 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:50.020964 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":412,"threads_started":1}
I20260927 21:42:50.485612 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:50.484833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:55935)
0927 21:42:50.485345 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:50.485426 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
I20260927 21:42:50.553619 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:50.552899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:56017)
0927 21:42:50.553352 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:50.553458 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":237,"threads_started":1}
I20260927 21:42:51.023073 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:51.022267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:33287)
0927 21:42:51.022797 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:51.022895 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
I20260927 21:42:51.487362 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:51.486594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:48889)
0927 21:42:51.487085 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:51.487183 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
I20260927 21:42:51.555639 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:51.554740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:52991)
0927 21:42:51.555246 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:51.555334 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":223,"threads_started":1}
I20260927 21:42:52.025338 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:52.024221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:39871)
0927 21:42:52.024940 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:52.025067 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":251,"threads_started":1}
I20260927 21:42:52.489650 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:52.488556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:37135)
0927 21:42:52.489336 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:52.489426 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":296,"threads_started":1}
I20260927 21:42:52.557624 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:52.556758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:44119)
0927 21:42:52.557291 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:52.557436 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":251,"threads_started":1}
I20260927 21:42:53.027099 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:53.026341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:53785)
0927 21:42:53.026841 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:53.026926 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260927 21:42:53.491585 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:53.490748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:52957)
0927 21:42:53.491315 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:53.491406 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":280,"threads_started":1}
I20260927 21:42:53.559571 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:53.558743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:34589)
0927 21:42:53.559283 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:53.559376 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":254,"threads_started":1}
I20260927 21:42:54.029199 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:54.028264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:53919)
0927 21:42:54.028899 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:54.029021 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":311,"threads_started":1}
I20260927 21:42:54.493566 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:54.492727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:45909)
0927 21:42:54.493270 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:54.493361 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":261,"threads_started":1}
I20260927 21:42:54.561712 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:54.560862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:35475)
0927 21:42:54.561445 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:54.561538 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":291,"threads_started":1}
I20260927 21:42:55.031023 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:55.030307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:56409)
0927 21:42:55.030754 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:55.030846 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
I20260927 21:42:55.495468 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:55.494682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:54675)
0927 21:42:55.495166 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:55.495253 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":270,"threads_started":1}
I20260927 21:42:55.563639 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:55.562793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:45307)
0927 21:42:55.563346 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:55.563452 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":240,"threads_started":1}
I20260927 21:42:56.033077 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:56.032199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:59073)
0927 21:42:56.032746 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:56.032859 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":308,"threads_started":1}
I20260927 21:42:56.497431 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:56.496620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:39357)
0927 21:42:56.497070 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:56.497189 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":219,"threads_started":1}
I20260927 21:42:56.565646 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:56.564762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:48675)
0927 21:42:56.565275 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:56.565445 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260927 21:42:57.035117 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:57.034228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:58109)
0927 21:42:57.034821 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:57.034940 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":273,"threads_started":1}
I20260927 21:42:57.499501 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:57.498609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:58855)
0927 21:42:57.499147 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:57.499253 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
I20260927 21:42:57.567691 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:57.566830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:39901)
0927 21:42:57.567373 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:57.567491 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":267,"threads_started":1}
I20260927 21:42:58.036908 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:58.036046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:56449)
0927 21:42:58.036598 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:58.036717 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":317,"threads_started":1}
I20260927 21:42:58.501437 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:58.500647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:43865)
0927 21:42:58.501150 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:58.501247 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260927 21:42:58.569636 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:58.568742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:52425)
0927 21:42:58.569305 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:58.569452 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":340,"threads_started":1}
I20260927 21:42:59.038753 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:59.037866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:36115)
0927 21:42:59.038461 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:59.038556 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":286,"threads_started":1}
I20260927 21:42:59.503175 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:59.502343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:47725)
0927 21:42:59.502857 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:59.502969 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":289,"threads_started":1}
I20260927 21:42:59.571650 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:42:59.570869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.129:57395)
0927 21:42:59.571361 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 21:42:59.571462 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260927 21:43:00.040850 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:00.039979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.130:35635)
0927 21:43:00.040533 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:00.040635 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":241,"threads_started":1}
I20260927 21:43:00.476184 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 15859
I20260927 21:43:00.505808 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:00.504758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35289 (local address 127.0.202.131:53481)
0927 21:43:00.505410 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:00.505563 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":254,"threads_started":1}
I20260927 21:43:00.507773 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:00.477903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.0.202.129:41717)
0927 21:43:00.478965 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:00.478979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:00.479019 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:43:00.480931 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:00.480943 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:00.481549 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:00.481877 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:00.483478 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:00.485057 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:00.486434 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:00.487204 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:00.487368 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:00.499178 (+ 11810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:00.499189 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:00.499208 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:00.499270 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:00.502777 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:00.504013 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:00.504026 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:00.504037 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:00.504373 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:00.505146 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:00.505153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:00.505156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:00.506129 (+ 973us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:00.506440 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:00.506944 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 21:43:00.507432 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":735,"threads_started":1}
I20260927 21:43:00.509351 16016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:00.596488 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 15859
I20260927 21:43:00.613804 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16024
I20260927 21:43:00.637959 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:00.615014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.0.202.130:41101)
0927 21:43:00.615685 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:00.615700 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:00.615739 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:43:00.617566 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:00.617577 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:00.618060 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:00.618493 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:00.620111 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:00.621590 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:00.622973 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:00.623704 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:00.623914 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:00.626912 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:00.626921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:00.626939 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:00.626982 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:00.631681 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:00.633217 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:00.633228 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:00.633238 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:00.633511 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:00.634019 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:00.634025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:00.634028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:00.635070 (+ 1042us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:00.635523 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:00.636849 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0927 21:43:00.637437 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":376,"threads_started":1}
I20260927 21:43:00.639850 16147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:00.726047 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16024
I20260927 21:43:00.744498 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16155
I20260927 21:43:00.764569 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:00.745741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.202.131:43879)
0927 21:43:00.746397 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:00.746417 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:00.746465 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:43:00.748109 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:00.748119 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:00.748848 (+ 729us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:00.749170 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:00.750536 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:00.752510 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:00.753916 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:00.754718 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:00.754872 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:00.757612 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:00.757621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:00.757639 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:00.757684 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:00.760973 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:00.761723 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:00.761731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:00.761738 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:00.761966 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:00.762420 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:00.762423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:00.762425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:00.762997 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:00.763237 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:00.763678 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 21:43:00.764136 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":310,"threads_started":1}
I20260927 21:43:00.766113 16278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:00.850797 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16155
2026-09-27T21:43:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24718 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:43:00 dist-test-slave-25dq krb5kdc[16341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:43:00 dist-test-slave-25dq krb5kdc[16341](info): set up 2 sockets
Sep 27 21:43:00 dist-test-slave-25dq krb5kdc[16341](info): commencing operation
krb5kdc: starting...
W20260927 21:43:01.997434 810 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:43:02 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:43:02.257681 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40549
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:02.602810 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:02.603209 16354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:02.603379 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:02.611263 16354 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:43:02.611406 16354 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:43:02.611500 16354 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:43:02.611589 16354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:43:02.611688 16354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:02.611807 16354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:02.611894 16354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:43:02.611974 16354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:43:02.622797 16354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:02.624218 16354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:02.625629 16354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:02.642433 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:02.642467 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:02.642529 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:02.642871 16354 server_base.cc:1034] running on GCE node
I20260927 21:43:02.643988 16354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:02.645793 16354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:02.647100 16354 hybrid_clock.cc:648] HybridClock initialized: now 1790545382647031 us; error 46 us; skew 500 ppm
I20260927 21:43:02.647580 16354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:02 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:02.651403 16354 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:43:02.653122 16354 webserver.cc:460] Webserver started at http://127.0.202.190:37579/ using document root <none> and password file <none>
I20260927 21:43:02.653940 16354 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:02.654103 16354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:02.654464 16354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:43:02.658123 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "ad7e6520141442caa1747bb8d02a2aee"
format_stamp: "Formatted at 2026-09-27 21:43:02 on dist-test-slave-25dq"
server_key: "b7bdf5b0bb2df3c7544c1c99b2dcf6cc"
server_key_iv: "1df39b67cc836ba213293c287b2e7891"
server_key_version: "encryptionkey@0"
I20260927 21:43:02.659050 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "ad7e6520141442caa1747bb8d02a2aee"
format_stamp: "Formatted at 2026-09-27 21:43:02 on dist-test-slave-25dq"
server_key: "b7bdf5b0bb2df3c7544c1c99b2dcf6cc"
server_key_iv: "1df39b67cc836ba213293c287b2e7891"
server_key_version: "encryptionkey@0"
I20260927 21:43:02.665033 16354 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 21:43:02.668970 16369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:02.670547 16354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:43:02.670758 16354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ad7e6520141442caa1747bb8d02a2aee"
format_stamp: "Formatted at 2026-09-27 21:43:02 on dist-test-slave-25dq"
server_key: "b7bdf5b0bb2df3c7544c1c99b2dcf6cc"
server_key_iv: "1df39b67cc836ba213293c287b2e7891"
server_key_version: "encryptionkey@0"
I20260927 21:43:02.671027 16354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:02.697005 16354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:02.701882 16354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:02.702198 16354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:02.722295 16354 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40549
I20260927 21:43:02.722419 16420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40549 every 8 connection(s)
I20260927 21:43:02.723766 16354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:43:02.727752 16421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:43:02.732985 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16354
I20260927 21:43:02.733242 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:43:02.734000 810 external_mini_cluster.cc:1428] Setting key 9d97df9a9107d9ed7e6636b398f6dce6
I20260927 21:43:02.739173 16421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Bootstrap starting.
I20260927 21:43:02.743299 16421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Neither blocks nor log segments found. Creating new log.
I20260927 21:43:02.744632 16421 log.cc:824] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:02.748005 16421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: No bootstrap required, opened a new log
Sep 27 21:43:02 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:43:02.757145 16421 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:02.757633 16421 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:43:02.757762 16421 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7e6520141442caa1747bb8d02a2aee, State: Initialized, Role: FOLLOWER
I20260927 21:43:02.758512 16421 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:02.758801 16421 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:43:02.759016 16421 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:43:02.759238 16421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:43:02.761538 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:02.736212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.0.202.190:40549)
0927 21:43:02.737359 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:02.737372 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:02.737416 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:43:02.739218 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:02.739242 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:02.739710 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:02.740084 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:02.742302 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:02.743919 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:02.745889 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:02.746705 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:02.746866 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:02.752258 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:02.752281 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:02.752354 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:02.752421 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:02.756419 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:02.757253 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:02.757265 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:02.757289 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:02.757481 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:02.757959 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:02.757964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:02.757967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:02.758700 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:02.759024 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:02.759554 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 21:43:02.760040 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":537,"threads_started":1}
I20260927 21:43:02.762710 16421 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:02.763278 16421 leader_election.cc:302] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad7e6520141442caa1747bb8d02a2aee; no voters:
I20260927 21:43:02.763965 16421 leader_election.cc:288] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:43:02.764091 16426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:43:02.765671 16426 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 LEADER]: Becoming Leader. State: Replica: ad7e6520141442caa1747bb8d02a2aee, State: Running, Role: LEADER
I20260927 21:43:02.766155 16421 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:43:02.766248 16426 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:02.770046 16428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad7e6520141442caa1747bb8d02a2aee. Latest consensus state: current_term: 1 leader_uuid: "ad7e6520141442caa1747bb8d02a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } } }
I20260927 21:43:02.770051 16427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad7e6520141442caa1747bb8d02a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } } }
I20260927 21:43:02.770601 16428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: This master's current role is: LEADER
I20260927 21:43:02.770709 16427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: This master's current role is: LEADER
I20260927 21:43:02.775082 16436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:43:02.781297 16436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:43:02.790395 16436 catalog_manager.cc:1347] Generated new cluster ID: 4d5074f665474e6fa8b8434686f0445a
I20260927 21:43:02.790525 16436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:43:02.812284 16436 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:43:02.813678 16436 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:43:02.826421 16436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Generated new TSK 0
I20260927 21:43:02.827401 16436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:43:02.893810 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40549
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:03.243659 16447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:03.243949 16447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:03.244096 16447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:03.251389 16447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:03.251504 16447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:03.251689 16447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:43:03.262568 16447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:03.264063 16447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:03.265444 16447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:03.279074 16452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:03.279140 16455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:03.279361 16453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:03.280212 16447 server_base.cc:1034] running on GCE node
I20260927 21:43:03.280911 16447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:03.282038 16447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:03.283309 16447 hybrid_clock.cc:648] HybridClock initialized: now 1790545383283257 us; error 78 us; skew 500 ppm
I20260927 21:43:03.283622 16447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:03 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:03.286917 16447 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:43:03.288697 16447 webserver.cc:460] Webserver started at http://127.0.202.129:40197/ using document root <none> and password file <none>
I20260927 21:43:03.289527 16447 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:03.289649 16447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:03.289975 16447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:43:03.293510 16447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "5a75efd563904395b97d09f0350077f2"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "9107b4191cfddbd01da7b31bfd1da282"
server_key_iv: "393dd8f0958f72182da4436f602c1ed1"
server_key_version: "encryptionkey@0"
I20260927 21:43:03.294313 16447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "5a75efd563904395b97d09f0350077f2"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "9107b4191cfddbd01da7b31bfd1da282"
server_key_iv: "393dd8f0958f72182da4436f602c1ed1"
server_key_version: "encryptionkey@0"
I20260927 21:43:03.300083 16447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:43:03.304114 16462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:03.305701 16447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:43:03.306002 16447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "5a75efd563904395b97d09f0350077f2"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "9107b4191cfddbd01da7b31bfd1da282"
server_key_iv: "393dd8f0958f72182da4436f602c1ed1"
server_key_version: "encryptionkey@0"
I20260927 21:43:03.306239 16447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:03.324067 16447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:03.328725 16447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:03.329067 16447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:03.330034 16447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:03.331560 16447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:03.331666 16447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:03.331827 16447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:03.331909 16447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:03.366016 16447 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:45275
I20260927 21:43:03.366096 16574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:45275 every 8 connection(s)
I20260927 21:43:03.367514 16447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:43:03.372326 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16447
I20260927 21:43:03.372548 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:43:03.373136 810 external_mini_cluster.cc:1428] Setting key bb2d9e3336d7f1fa378d9931d73788a8
Sep 27 21:43:03 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:43:03.391299 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:03.370144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:47379 (local address 127.0.202.190:40549)
0927 21:43:03.370587 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:03.370594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:03.371697 (+ 1103us) server_negotiation.cc:407] Connection header received
0927 21:43:03.373768 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:03.373772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:03.373846 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:03.373972 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:03.375756 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.376977 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.378779 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.379485 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.379633 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:03.383767 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:03.383772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:03.383776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:03.383814 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:03.387506 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:03.388422 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:03.388430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:03.388437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:03.388590 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:03.389102 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:03.389110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:03.389116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:03.389533 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:03.389692 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:03.390492 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 21:43:03.390775 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":214,"threads_started":1}
I20260927 21:43:03.392066 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:03.370761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:47379)
0927 21:43:03.371532 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:03.371570 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:03.373463 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:03.374202 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:03.374210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:03.374778 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:43:03.375580 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:03.375601 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.377180 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.377185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:03.378620 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:03.378635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.379729 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.379734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:03.379985 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:03.380851 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:43:03.380878 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:43:03.383498 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:43:03.387693 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:03.387701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:03.387725 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:03.388225 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:03.388732 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:03.388736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:03.388738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:03.388970 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:03.389678 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:43:03.389685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:43:03.390075 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 21:43:03.390342 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 21:43:03.390767 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":291,"threads_started":1}
I20260927 21:43:03.394001 16575 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40549
I20260927 21:43:03.394375 16575 heartbeater.cc:464] Registering TS with master...
I20260927 21:43:03.395314 16575 heartbeater.cc:511] Master 127.0.202.190:40549 requested a full tablet report, sending...
I20260927 21:43:03.397962 16386 ts_manager.cc:194] Registered new tserver with Master: 5a75efd563904395b97d09f0350077f2 (127.0.202.129:45275)
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:43:03.399616 16386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:47379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:43:03.432940 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40549
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:03.773301 16581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:03.773643 16581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:03.773813 16581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:03.781332 16581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:03.781512 16581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:03.781687 16581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:43:03.791855 16581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:03.793413 16581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:03.794773 16581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:03.808665 16589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:03.808691 16587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:03.808779 16586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:03.809566 16581 server_base.cc:1034] running on GCE node
I20260927 21:43:03.810312 16581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:03.811443 16581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:03.812692 16581 hybrid_clock.cc:648] HybridClock initialized: now 1790545383812621 us; error 62 us; skew 500 ppm
I20260927 21:43:03.813092 16581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:03 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:03.816736 16581 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:43:03.818567 16581 webserver.cc:460] Webserver started at http://127.0.202.130:44131/ using document root <none> and password file <none>
I20260927 21:43:03.819339 16581 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:03.819502 16581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:03.819873 16581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:43:03.823203 16581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "697123d814da43748e51e516cff01a47"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "317172484166d947cb486935f5edc643"
server_key_iv: "70a69ae21d0b3ebb118955785ebfd399"
server_key_version: "encryptionkey@0"
I20260927 21:43:03.824028 16581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "697123d814da43748e51e516cff01a47"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "317172484166d947cb486935f5edc643"
server_key_iv: "70a69ae21d0b3ebb118955785ebfd399"
server_key_version: "encryptionkey@0"
I20260927 21:43:03.829959 16581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 21:43:03.833550 16596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:03.835106 16581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:43:03.835306 16581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "697123d814da43748e51e516cff01a47"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "317172484166d947cb486935f5edc643"
server_key_iv: "70a69ae21d0b3ebb118955785ebfd399"
server_key_version: "encryptionkey@0"
I20260927 21:43:03.835602 16581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:03.860759 16581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:03.865481 16581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:03.865799 16581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:03.866962 16581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:03.868494 16581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:03.868566 16581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:03.868706 16581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:03.868757 16581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:03.906103 16581 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:39303
I20260927 21:43:03.906216 16708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:39303 every 8 connection(s)
I20260927 21:43:03.907600 16581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:43:03.917968 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16581
I20260927 21:43:03.918172 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:43:03.918807 810 external_mini_cluster.cc:1428] Setting key 1b5b58626b4cf36de162431fdfc7ec69
Sep 27 21:43:03 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:43:03.933136 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:03.910608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:54987 (local address 127.0.202.190:40549)
0927 21:43:03.911182 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:03.911189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:03.912538 (+ 1349us) server_negotiation.cc:407] Connection header received
0927 21:43:03.914524 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:03.914529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:03.914602 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:03.914708 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:03.916472 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.917279 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.919832 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.920597 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.920793 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:03.925413 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:03.925422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:03.925430 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:03.925482 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:03.929395 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:03.930514 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:03.930518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:03.930520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:03.930651 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:03.931215 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:03.931218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:03.931219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:03.931494 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:03.931605 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:03.932654 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0927 21:43:03.932832 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":435,"thread_start_us":280,"threads_started":1}
I20260927 21:43:03.934916 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:03.911387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:54987)
0927 21:43:03.912285 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:03.912401 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:03.914214 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:03.914967 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:03.914976 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:03.915582 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:43:03.916205 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:03.916223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.917501 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.917531 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:03.919593 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:03.919605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:03.920822 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:03.920828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:03.921089 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:03.921940 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:43:03.921977 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:43:03.925112 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:43:03.929593 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:03.929602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:03.929624 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:03.930347 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:03.930775 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:03.930784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:03.930789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:03.931089 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:03.931633 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:43:03.931645 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:43:03.932171 (+ 526us) client_negotiation.cc:769] Sending connection context
0927 21:43:03.932561 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 21:43:03.933227 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":387,"threads_started":1}
I20260927 21:43:03.936568 16709 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40549
I20260927 21:43:03.936932 16709 heartbeater.cc:464] Registering TS with master...
I20260927 21:43:03.937698 16709 heartbeater.cc:511] Master 127.0.202.190:40549 requested a full tablet report, sending...
I20260927 21:43:03.939275 16386 ts_manager.cc:194] Registered new tserver with Master: 697123d814da43748e51e516cff01a47 (127.0.202.130:39303)
I20260927 21:43:03.940146 16386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:54987
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:43:03.981535 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40549
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:04.315654 16715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:04.315953 16715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:04.316123 16715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:04.323408 16715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:04.323519 16715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:04.323702 16715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:43:04.333515 16715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:04.335095 16715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:04.336611 16715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:04.351008 16720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:04.351109 16721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:04.351109 16723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:04.352157 16715 server_base.cc:1034] running on GCE node
I20260927 21:43:04.352880 16715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:04.354060 16715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:04.355391 16715 hybrid_clock.cc:648] HybridClock initialized: now 1790545384355400 us; error 135 us; skew 500 ppm
I20260927 21:43:04.355751 16715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:04 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:04.359335 16715 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:43:04.361258 16715 webserver.cc:460] Webserver started at http://127.0.202.131:46825/ using document root <none> and password file <none>
I20260927 21:43:04.362022 16715 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:04.362191 16715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:04.362531 16715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:43:04.366011 16715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "b1a488699b9645a89e834a24d3c60b1d"
format_stamp: "Formatted at 2026-09-27 21:43:04 on dist-test-slave-25dq"
server_key: "665357b12d6bd037c2288c6b4829db9a"
server_key_iv: "5f3288b96360139aef73ff1a7610aa23"
server_key_version: "encryptionkey@0"
I20260927 21:43:04.366875 16715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "b1a488699b9645a89e834a24d3c60b1d"
format_stamp: "Formatted at 2026-09-27 21:43:04 on dist-test-slave-25dq"
server_key: "665357b12d6bd037c2288c6b4829db9a"
server_key_iv: "5f3288b96360139aef73ff1a7610aa23"
server_key_version: "encryptionkey@0"
I20260927 21:43:04.372978 16715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 21:43:04.377034 16730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:04.378499 16715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:43:04.378760 16715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "b1a488699b9645a89e834a24d3c60b1d"
format_stamp: "Formatted at 2026-09-27 21:43:04 on dist-test-slave-25dq"
server_key: "665357b12d6bd037c2288c6b4829db9a"
server_key_iv: "5f3288b96360139aef73ff1a7610aa23"
server_key_version: "encryptionkey@0"
I20260927 21:43:04.378999 16715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:04.397073 16715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:04.401675 16715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:04.401995 16715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:04.403168 16715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:04.403443 16575 heartbeater.cc:503] Master 127.0.202.190:40549 was elected leader, sending a full tablet report...
I20260927 21:43:04.404958 16715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:04.405084 16715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:04.405247 16715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:04.405324 16715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:04.441341 16715 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37005
I20260927 21:43:04.441437 16842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37005 every 8 connection(s)
I20260927 21:43:04.442699 16715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:43:04.445995 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16715
I20260927 21:43:04.446233 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:43:04.446812 810 external_mini_cluster.cc:1428] Setting key 4c797d9b0741fa1de802a6416203f1b0
Sep 27 21:43:04 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:43:04.465091 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.445367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:55911 (local address 127.0.202.190:40549)
0927 21:43:04.445929 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.445934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.447370 (+ 1436us) server_negotiation.cc:407] Connection header received
0927 21:43:04.449172 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.449178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.449266 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.449395 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:04.451065 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.451846 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.453478 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.453893 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.454041 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.457861 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:04.457870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:04.457880 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:04.457920 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:04.461204 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.462262 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.462269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.462273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.462430 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.463008 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.463014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.463024 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.463335 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:04.463530 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.464444 (+ 914us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.464691 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":299,"threads_started":1}
I20260927 21:43:04.466220 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.445893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:55911)
0927 21:43:04.446773 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:04.446813 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:04.448888 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:04.449646 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:04.449655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:04.450317 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:43:04.450897 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.450915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.452016 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.452020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.453332 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.453344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.454122 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.454129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.454372 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.455100 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:43:04.455128 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:43:04.457606 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:43:04.461391 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:04.461406 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:04.461423 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:04.462109 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:04.462573 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:04.462582 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:04.462585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:04.462860 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:04.463480 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:43:04.463490 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:43:04.463897 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 21:43:04.464210 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 21:43:04.464921 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":410,"threads_started":1}
I20260927 21:43:04.467796 16843 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40549
I20260927 21:43:04.468151 16843 heartbeater.cc:464] Registering TS with master...
I20260927 21:43:04.468971 16843 heartbeater.cc:511] Master 127.0.202.190:40549 requested a full tablet report, sending...
I20260927 21:43:04.470818 16386 ts_manager.cc:194] Registered new tserver with Master: b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005)
I20260927 21:43:04.471647 16386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:55911
I20260927 21:43:04.475314 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:43:04.494479 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.480073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.202.190:40549)
0927 21:43:04.480246 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.480250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.480670 (+ 420us) server_negotiation.cc:407] Connection header received
0927 21:43:04.480798 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.480801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.480881 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.480972 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:04.482709 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.483574 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.485170 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.485651 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.485765 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.488231 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:04.488236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:04.488239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:04.488283 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:04.491145 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.492250 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.492254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.492257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.492578 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.492975 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.492979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.492982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.493393 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:04.493627 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.493989 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.494191 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:43:04.498958 16386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:43:04.506047 16386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:43:04.542081 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.520449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.0.202.130:39303)
0927 21:43:04.532104 (+ 11655us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.532115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.533034 (+ 919us) server_negotiation.cc:407] Connection header received
0927 21:43:04.533245 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.533252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.533513 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.533704 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:04.535824 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.537237 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.539629 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.540782 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.540948 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.541178 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.541511 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.541713 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11416,"thread_start_us":11658,"threads_started":1}
I20260927 21:43:04.542495 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.520246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.0.202.129:45275)
0927 21:43:04.532075 (+ 11829us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.532088 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.533121 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 21:43:04.533232 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.533240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.533514 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.533704 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:04.534887 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.536542 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.539696 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.541188 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.541308 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.541529 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.541759 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.541938 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11621,"thread_start_us":11412,"threads_started":1}
I20260927 21:43:04.543433 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.520176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39303 (local address 127.0.0.1:60890)
0927 21:43:04.532761 (+ 12585us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:04.532783 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:04.533001 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:04.534012 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:04.534021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:04.534074 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:04.535605 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.535621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.537381 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.537386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.539463 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.539477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.541004 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.541011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.541195 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.541259 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 21:43:04.542490 (+ 1231us) client_negotiation.cc:240] Negotiation successful
0927 21:43:04.542710 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12432,"mutex_wait_us":1124,"spinlock_wait_cycles":28416,"thread_start_us":12277,"threads_started":1}
I20260927 21:43:04.543596 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.520154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:45275 (local address 127.0.0.1:58094)
0927 21:43:04.532692 (+ 12538us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:04.532746 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:04.533002 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:04.534012 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:04.534020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:04.534061 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:04.534610 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.534628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.536686 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.536693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.539415 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.539430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.541377 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.541386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.541564 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.541596 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:43:04.542985 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0927 21:43:04.543165 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12188,"mutex_wait_us":58,"spinlock_wait_cycles":10624,"thread_start_us":11967,"threads_started":1}
I20260927 21:43:04.544072 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.533477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:37005 (local address 127.0.0.1:57748)
0927 21:43:04.535001 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:04.535021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:04.535128 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:04.536253 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:04.536258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:04.536286 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:04.536687 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.536701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.538656 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.538661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.540918 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.540938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.543063 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.543069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.543250 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.543308 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:43:04.543423 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:43:04.543680 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"mutex_wait_us":3,"thread_start_us":1238,"threads_started":1}
I20260927 21:43:04.544100 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.533461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.202.131:37005)
0927 21:43:04.535666 (+ 2205us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.535675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.535708 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:43:04.535801 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.535808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.536052 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.536239 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:04.536928 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.538457 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.541122 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.542537 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.542692 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.542930 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.543525 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.543721 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2035,"thread_start_us":1893,"threads_started":1}
I20260927 21:43:04.547981 16510 tablet_service.cc:1467] Processing CreateTablet for tablet 618fecf6c159470eb80e9082933bc658 (DEFAULT_TABLE table=test-table [id=5ceaeefeadb64992bac62582fe11f2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:43:04.547988 16778 tablet_service.cc:1467] Processing CreateTablet for tablet 618fecf6c159470eb80e9082933bc658 (DEFAULT_TABLE table=test-table [id=5ceaeefeadb64992bac62582fe11f2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:43:04.547988 16644 tablet_service.cc:1467] Processing CreateTablet for tablet 618fecf6c159470eb80e9082933bc658 (DEFAULT_TABLE table=test-table [id=5ceaeefeadb64992bac62582fe11f2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:43:04.549413 16510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618fecf6c159470eb80e9082933bc658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:43:04.549445 16644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618fecf6c159470eb80e9082933bc658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:43:04.549445 16778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618fecf6c159470eb80e9082933bc658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:43:04.562455 16862 tablet_bootstrap.cc:492] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Bootstrap starting.
I20260927 21:43:04.564800 16864 tablet_bootstrap.cc:492] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Bootstrap starting.
I20260927 21:43:04.567310 16862 tablet_bootstrap.cc:654] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Neither blocks nor log segments found. Creating new log.
I20260927 21:43:04.567550 16863 tablet_bootstrap.cc:492] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Bootstrap starting.
I20260927 21:43:04.568938 16864 tablet_bootstrap.cc:654] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Neither blocks nor log segments found. Creating new log.
I20260927 21:43:04.568971 16862 log.cc:824] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:04.570335 16864 log.cc:824] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:04.571287 16863 tablet_bootstrap.cc:654] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Neither blocks nor log segments found. Creating new log.
I20260927 21:43:04.572500 16863 log.cc:824] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:04.572732 16862 tablet_bootstrap.cc:492] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: No bootstrap required, opened a new log
I20260927 21:43:04.573082 16862 ts_tablet_manager.cc:1397] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 21:43:04.574010 16864 tablet_bootstrap.cc:492] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: No bootstrap required, opened a new log
I20260927 21:43:04.574294 16864 ts_tablet_manager.cc:1397] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 21:43:04.584415 16862 raft_consensus.cc:348] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.584931 16862 raft_consensus.cc:374] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:43:04.585072 16862 raft_consensus.cc:729] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a75efd563904395b97d09f0350077f2, State: Initialized, Role: FOLLOWER
I20260927 21:43:04.585124 16863 tablet_bootstrap.cc:492] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: No bootstrap required, opened a new log
I20260927 21:43:04.585143 16864 raft_consensus.cc:348] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.585457 16863 ts_tablet_manager.cc:1397] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260927 21:43:04.585671 16864 raft_consensus.cc:374] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:43:04.585813 16864 raft_consensus.cc:729] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697123d814da43748e51e516cff01a47, State: Initialized, Role: FOLLOWER
I20260927 21:43:04.585884 16862 consensus_queue.cc:260] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.586608 16864 consensus_queue.cc:260] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.587963 16862 ts_tablet_manager.cc:1428] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 21:43:04.589077 16709 heartbeater.cc:503] Master 127.0.202.190:40549 was elected leader, sending a full tablet report...
I20260927 21:43:04.589704 16864 ts_tablet_manager.cc:1428] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260927 21:43:04.596345 16863 raft_consensus.cc:348] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.596774 16863 raft_consensus.cc:374] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:43:04.596902 16863 raft_consensus.cc:729] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a488699b9645a89e834a24d3c60b1d, State: Initialized, Role: FOLLOWER
I20260927 21:43:04.597466 16863 consensus_queue.cc:260] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.598531 16843 heartbeater.cc:503] Master 127.0.202.190:40549 was elected leader, sending a full tablet report...
I20260927 21:43:04.599110 16863 ts_tablet_manager.cc:1428] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260927 21:43:04.619993 16576 tablet.cc:2369] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:43:04.637051 16868 raft_consensus.cc:482] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:43:04.637483 16868 raft_consensus.cc:504] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.639987 16868 leader_election.cc:288] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 697123d814da43748e51e516cff01a47 (127.0.202.130:39303), b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005)
I20260927 21:43:04.647994 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.640475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:40401 (local address 127.0.202.130:39303)
0927 21:43:04.640734 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.640742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.641186 (+ 444us) server_negotiation.cc:407] Connection header received
0927 21:43:04.641476 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.641481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.641590 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.641678 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:04.642425 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.643463 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.645576 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.646963 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.647111 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.647172 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.647625 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.647769 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:43:04.648123 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.640412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39303 (local address 127.0.202.129:40401)
0927 21:43:04.641058 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:04.641097 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:04.641323 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:04.641863 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:04.641867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:04.641892 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:04.642247 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.642259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.643620 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.643626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.645406 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.645418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.647223 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.647231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.647420 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.647459 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:43:04.647575 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:43:04.647724 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":356,"threads_started":1}
I20260927 21:43:04.649277 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.641478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:38827 (local address 127.0.202.131:37005)
0927 21:43:04.641718 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.641725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.642207 (+ 482us) server_negotiation.cc:407] Connection header received
0927 21:43:04.642358 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.642364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.642450 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.642533 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:04.643190 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.644253 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.646795 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.648145 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.648376 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.648456 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.648899 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.649030 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:43:04.649353 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.641152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:37005 (local address 127.0.202.129:38827)
0927 21:43:04.642085 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:04.642105 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:04.642248 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:04.642588 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:04.642593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:04.642616 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:04.642986 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.642997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.644468 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.644473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.646580 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:04.646599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.648603 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.648608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:04.648723 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.648744 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:43:04.648862 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 21:43:04.648993 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":1,"thread_start_us":544,"threads_started":1}
I20260927 21:43:04.649217 16664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618fecf6c159470eb80e9082933bc658" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697123d814da43748e51e516cff01a47" is_pre_election: true
I20260927 21:43:04.649828 16664 raft_consensus.cc:2454] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 0.
I20260927 21:43:04.650593 16798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618fecf6c159470eb80e9082933bc658" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a488699b9645a89e834a24d3c60b1d" is_pre_election: true
I20260927 21:43:04.650701 16466 leader_election.cc:302] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a75efd563904395b97d09f0350077f2, 697123d814da43748e51e516cff01a47; no voters:
I20260927 21:43:04.651168 16798 raft_consensus.cc:2454] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 0.
I20260927 21:43:04.651204 16868 raft_consensus.cc:2781] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:43:04.651386 16868 raft_consensus.cc:482] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:43:04.651508 16868 raft_consensus.cc:3037] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:43:04.654052 16868 raft_consensus.cc:504] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.654719 16868 leader_election.cc:288] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 election: Requested vote from peers 697123d814da43748e51e516cff01a47 (127.0.202.130:39303), b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005)
I20260927 21:43:04.655316 16664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618fecf6c159470eb80e9082933bc658" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697123d814da43748e51e516cff01a47"
I20260927 21:43:04.655361 16798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618fecf6c159470eb80e9082933bc658" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a488699b9645a89e834a24d3c60b1d"
I20260927 21:43:04.655541 16664 raft_consensus.cc:3037] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:43:04.655615 16798 raft_consensus.cc:3037] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:43:04.657995 16798 raft_consensus.cc:2454] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 1.
I20260927 21:43:04.658077 16664 raft_consensus.cc:2454] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 1.
I20260927 21:43:04.658511 16466 leader_election.cc:302] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a75efd563904395b97d09f0350077f2, b1a488699b9645a89e834a24d3c60b1d; no voters:
I20260927 21:43:04.658929 16868 raft_consensus.cc:2781] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:43:04.659505 16868 raft_consensus.cc:686] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5a75efd563904395b97d09f0350077f2, State: Running, Role: LEADER
W20260927 21:43:04.659924 16710 tablet.cc:2369] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:43:04.660580 16868 consensus_queue.cc:237] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } }
I20260927 21:43:04.666041 16385 catalog_manager.cc:5495] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a75efd563904395b97d09f0350077f2 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "5a75efd563904395b97d09f0350077f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } health_report { overall_health: UNKNOWN } } }
W20260927 21:43:04.694368 16844 tablet.cc:2369] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:43:04.705665 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.699995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.0.202.129:45275)
0927 21:43:04.700200 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.700205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.700223 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:43:04.700429 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.700434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.700533 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.700657 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:43:04.701035 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.701849 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.703363 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.703766 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.703860 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.704469 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:43:04.705019 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:43:04.705130 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.705248 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.705383 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:43:04.715627 16798 raft_consensus.cc:1261] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Refusing update from remote peer 5a75efd563904395b97d09f0350077f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:43:04.715652 16664 raft_consensus.cc:1261] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Refusing update from remote peer 5a75efd563904395b97d09f0350077f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:43:04.716910 16873 consensus_queue.cc:1035] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:43:04.717396 16868 consensus_queue.cc:1035] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:43:04.731305 16877 mvcc.cc:205] Tried to move back new op lower bound from 7334073895782567936 to 7334073895574765568. Current Snapshot: MvccSnapshot[applied={T|T < 7334073895782567936}]
I20260927 21:43:04.757411 16386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:43:04.757767 16386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:43:04.762575 16386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Sending DeleteTablet for 3 replicas of tablet 618fecf6c159470eb80e9082933bc658
I20260927 21:43:04.764101 16644 tablet_service.cc:1514] Processing DeleteTablet for tablet 618fecf6c159470eb80e9082933bc658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:43:04 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60890
I20260927 21:43:04.764125 16778 tablet_service.cc:1514] Processing DeleteTablet for tablet 618fecf6c159470eb80e9082933bc658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:43:04 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:57748
I20260927 21:43:04.764125 16510 tablet_service.cc:1514] Processing DeleteTablet for tablet 618fecf6c159470eb80e9082933bc658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:43:04 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:58094
I20260927 21:43:04.765797 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16447
I20260927 21:43:04.776858 16886 tablet_replica.cc:330] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: stopping tablet replica
I20260927 21:43:04.777558 16886 raft_consensus.cc:2229] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:43:04.778259 16886 raft_consensus.cc:2258] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:43:04.779594 16885 tablet_replica.cc:330] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: stopping tablet replica
I20260927 21:43:04.780180 16885 raft_consensus.cc:2229] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:43:04.780894 16885 raft_consensus.cc:2258] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:43:04.783772 16887 tablet_replica.cc:330] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: stopping tablet replica
I20260927 21:43:04.784451 16887 raft_consensus.cc:2229] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:43:04.785291 16887 raft_consensus.cc:2258] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:43:04.785841 16885 ts_tablet_manager.cc:1905] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:43:04.787811 16887 ts_tablet_manager.cc:1905] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:43:04.792210 16886 ts_tablet_manager.cc:1905] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:43:04 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:43:04.795957 16887 ts_tablet_manager.cc:1918] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:43:04.796164 16887 log.cc:1196] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/618fecf6c159470eb80e9082933bc658
I20260927 21:43:04.796804 16887 ts_tablet_manager.cc:1939] T 618fecf6c159470eb80e9082933bc658 P 5a75efd563904395b97d09f0350077f2: Deleting consensus metadata
I20260927 21:43:04.799191 16370 catalog_manager.cc:4841] TS 5a75efd563904395b97d09f0350077f2 (127.0.202.129:45275): tablet 618fecf6c159470eb80e9082933bc658 (table test-table [id=5ceaeefeadb64992bac62582fe11f2af]) successfully deleted
I20260927 21:43:04.799818 16886 ts_tablet_manager.cc:1918] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:43:04.800007 16886 log.cc:1196] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/618fecf6c159470eb80e9082933bc658
I20260927 21:43:04.802125 16885 ts_tablet_manager.cc:1918] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:43:04.802312 16885 log.cc:1196] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/618fecf6c159470eb80e9082933bc658
I20260927 21:43:04.802431 16886 ts_tablet_manager.cc:1939] T 618fecf6c159470eb80e9082933bc658 P b1a488699b9645a89e834a24d3c60b1d: Deleting consensus metadata
I20260927 21:43:04.802906 16885 ts_tablet_manager.cc:1939] T 618fecf6c159470eb80e9082933bc658 P 697123d814da43748e51e516cff01a47: Deleting consensus metadata
I20260927 21:43:04.804281 16373 catalog_manager.cc:4841] TS b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005): tablet 618fecf6c159470eb80e9082933bc658 (table test-table [id=5ceaeefeadb64992bac62582fe11f2af]) successfully deleted
I20260927 21:43:04.804628 16373 catalog_manager.cc:4841] TS 697123d814da43748e51e516cff01a47 (127.0.202.130:39303): tablet 618fecf6c159470eb80e9082933bc658 (table test-table [id=5ceaeefeadb64992bac62582fe11f2af]) successfully deleted
I20260927 21:43:04.806613 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.767086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.0.202.129:45275)
0927 21:43:04.767288 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.767295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.784019 (+ 16724us) server_negotiation.cc:407] Connection header received
0927 21:43:04.784438 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.784446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.784541 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.784640 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:04.786862 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.787990 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.790097 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.790830 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.790968 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.796969 (+ 6001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:04.796986 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:04.797008 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:04.797079 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:04.801249 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.802091 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.802105 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.802111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.802454 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.804867 (+ 2413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.804873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.804876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.805353 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:04.805720 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.805981 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.806240 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:43:04.807598 16570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:04.909138 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16447
I20260927 21:43:04.927904 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16581
Sep 27 21:43:04 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:43:04.945333 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:04.928766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.0.202.130:39303)
0927 21:43:04.928948 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:04.928953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:04.928971 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:43:04.929102 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:04.929106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:04.929174 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:04.929256 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:04.930635 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.931408 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.932913 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:04.933346 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:04.933466 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:04.939024 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:04.939042 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:04.939059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:04.939104 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:04.942299 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.943046 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.943054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.943056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.943264 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:04.943648 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:04.943651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:04.943653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:04.944054 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:04.944434 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:04.944657 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:43:04.944840 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:43:04.946100 16704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:05.039865 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16581
I20260927 21:43:05.058516 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16715
Sep 27 21:43:05 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:43:05.075888 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:05.059415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.202.131:37005)
0927 21:43:05.059591 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:05.059596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:05.059615 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:43:05.059777 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:05.059782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:05.059857 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:05.059948 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:05.061353 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:05.062109 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:05.063672 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:05.064141 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:05.064414 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:05.068997 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:05.069015 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:05.069032 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:05.069071 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:05.072689 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:05.073532 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:05.073540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:05.073543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:05.073710 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:05.074067 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:05.074072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:05.074078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:05.074580 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:05.074822 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:05.075294 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 21:43:05.075538 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:43:05.076675 16838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:05.169430 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16715
I20260927 21:43:05.188669 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16354
I20260927 21:43:05.189939 16416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:05.306663 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16354
I20260927 21:43:25.329293 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40549
--webserver_interface=127.0.202.190
--webserver_port=37579
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:25.674059 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:25.674424 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:25.674575 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:25.682090 16893 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:43:25.682238 16893 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:43:25.682335 16893 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:43:25.682420 16893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:43:25.682515 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:25.682638 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:25.682725 16893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:43:25.682803 16893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:43:25.692894 16893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=37579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:25.694262 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:25.695689 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:25.720562 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:25.720656 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:25.720593 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:25.721489 16893 server_base.cc:1034] running on GCE node
I20260927 21:43:25.722478 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:25.724346 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:25.725656 16893 hybrid_clock.cc:648] HybridClock initialized: now 1790545405725562 us; error 66 us; skew 500 ppm
I20260927 21:43:25.726148 16893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:25 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:25.737272 16893 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:43:25.739346 16893 webserver.cc:460] Webserver started at http://127.0.202.190:37579/ using document root <none> and password file <none>
I20260927 21:43:25.740276 16893 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:25.740506 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:25.747717 16893 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 21:43:25.751559 16909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:25.753463 16893 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 21:43:25.753715 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ad7e6520141442caa1747bb8d02a2aee"
format_stamp: "Formatted at 2026-09-27 21:43:02 on dist-test-slave-25dq"
server_key: "b7bdf5b0bb2df3c7544c1c99b2dcf6cc"
server_key_iv: "1df39b67cc836ba213293c287b2e7891"
server_key_version: "encryptionkey@0"
I20260927 21:43:25.755000 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:25.811723 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:25.816910 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:25.817286 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:25.839619 16893 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40549
I20260927 21:43:25.839668 16960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40549 every 8 connection(s)
I20260927 21:43:25.841370 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:43:25.846405 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16893
I20260927 21:43:25.847477 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:45275
--local_ip_for_outbound_sockets=127.0.202.129
--tserver_master_addrs=127.0.202.190:40549
--webserver_port=40197
--webserver_interface=127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:43:25.849717 16961 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:43:25.855288 16961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Bootstrap starting.
I20260927 21:43:25.877099 16961 log.cc:824] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:25.890576 16961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:43:25.891192 16961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Bootstrap complete.
I20260927 21:43:25.900934 16961 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:25.901960 16961 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7e6520141442caa1747bb8d02a2aee, State: Initialized, Role: FOLLOWER
I20260927 21:43:25.902719 16961 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:25.903000 16961 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:43:25.903147 16961 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:43:25.903337 16961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:43:25.908038 16961 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:25.908643 16961 leader_election.cc:302] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad7e6520141442caa1747bb8d02a2aee; no voters:
I20260927 21:43:25.909322 16961 leader_election.cc:288] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:43:25.909468 16965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:43:25.911911 16961 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:43:25.916637 16965 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 LEADER]: Becoming Leader. State: Replica: ad7e6520141442caa1747bb8d02a2aee, State: Running, Role: LEADER
I20260927 21:43:25.917418 16965 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:25.920554 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad7e6520141442caa1747bb8d02a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } } }
I20260927 21:43:25.921070 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: This master's current role is: LEADER
I20260927 21:43:25.922010 16975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:43:25.921813 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad7e6520141442caa1747bb8d02a2aee. Latest consensus state: current_term: 2 leader_uuid: "ad7e6520141442caa1747bb8d02a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } } }
I20260927 21:43:25.922911 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: This master's current role is: LEADER
I20260927 21:43:25.941026 16975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:43:25.943003 16975 catalog_manager.cc:1259] Loaded cluster ID: 4d5074f665474e6fa8b8434686f0445a
I20260927 21:43:25.943208 16975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:43:25.948949 16975 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:43:25.951097 16975 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Loaded TSK: 0
I20260927 21:43:25.974543 16983 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:43:25.998111 16975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Generated new TSK 1
I20260927 21:43:25.998565 16975 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:43:26.238600 16963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:26.238950 16963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:26.239125 16963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:26.246752 16963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:26.246927 16963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:26.247109 16963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:43:26.258232 16963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:45275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=40197
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:26.259652 16963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:26.261193 16963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:26.276732 16990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:26.277313 16992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:26.277487 16989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:26.277437 16963 server_base.cc:1034] running on GCE node
I20260927 21:43:26.278434 16963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:26.279623 16963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:26.280946 16963 hybrid_clock.cc:648] HybridClock initialized: now 1790545406280897 us; error 36 us; skew 500 ppm
I20260927 21:43:26.281365 16963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:26 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:26.285162 16963 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:43:26.287191 16963 webserver.cc:460] Webserver started at http://127.0.202.129:40197/ using document root <none> and password file <none>
I20260927 21:43:26.288178 16963 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:26.288369 16963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:26.296137 16963 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 21:43:26.299814 16999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:26.301574 16963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:43:26.301750 16963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "5a75efd563904395b97d09f0350077f2"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "9107b4191cfddbd01da7b31bfd1da282"
server_key_iv: "393dd8f0958f72182da4436f602c1ed1"
server_key_version: "encryptionkey@0"
I20260927 21:43:26.303143 16963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:26.336666 16963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:26.341288 16963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:26.341624 16963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:26.342782 16963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:26.344275 16963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:26.344424 16963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 21:43:26.344547 16963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:26.344645 16963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:26.381232 16963 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:45275
I20260927 21:43:26.381376 17111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:45275 every 8 connection(s)
I20260927 21:43:26.382711 16963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:43:26.386931 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 16963
I20260927 21:43:26.387611 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:39303
--local_ip_for_outbound_sockets=127.0.202.130
--tserver_master_addrs=127.0.202.190:40549
--webserver_port=44131
--webserver_interface=127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:43:26 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:43:26.408665 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:26.385626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:52241 (local address 127.0.202.190:40549)
0927 21:43:26.386225 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:26.386243 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:26.386913 (+ 670us) server_negotiation.cc:407] Connection header received
0927 21:43:26.389285 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:26.389305 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:26.389784 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:26.390178 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:26.391638 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.393761 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.395778 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.396524 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.396734 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:26.400482 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:26.400491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:26.400507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:26.400550 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:26.403733 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:26.404660 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:26.404668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:26.404674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:26.404838 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:26.405318 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:26.405321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:26.405324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:26.405835 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:26.406042 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:26.406857 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 21:43:26.407250 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
I20260927 21:43:26.408658 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:26.385872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:52241)
0927 21:43:26.386700 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:26.386754 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:26.388695 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:26.390294 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:26.390301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:26.390852 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:43:26.391397 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:26.391414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.393955 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.393963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:26.395619 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:26.395632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.396787 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.396794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:26.397074 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:26.397773 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:43:26.397796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:43:26.400170 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:43:26.403910 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:26.403920 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:26.403937 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:26.404505 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:26.404973 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:26.404977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:26.404981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:26.405186 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:26.405994 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:43:26.406002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:43:26.406352 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 21:43:26.406619 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 21:43:26.407206 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":351,"threads_started":1}
I20260927 21:43:26.412199 17112 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40549
I20260927 21:43:26.412578 17112 heartbeater.cc:464] Registering TS with master...
I20260927 21:43:26.413425 17112 heartbeater.cc:511] Master 127.0.202.190:40549 requested a full tablet report, sending...
I20260927 21:43:26.415876 16926 ts_manager.cc:194] Registered new tserver with Master: 5a75efd563904395b97d09f0350077f2 (127.0.202.129:45275)
I20260927 21:43:26.418022 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:52241
W20260927 21:43:26.737651 17116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:26.737946 17116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:26.738042 17116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:26.745497 17116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:26.745627 17116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:26.745743 17116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:43:26.755686 17116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:39303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=44131
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:26.757105 17116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:26.758349 17116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:26.772325 17121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:26.772315 17124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:26.772369 17122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:26.773218 17116 server_base.cc:1034] running on GCE node
I20260927 21:43:26.773811 17116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:26.774833 17116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:26.776041 17116 hybrid_clock.cc:648] HybridClock initialized: now 1790545406775967 us; error 61 us; skew 500 ppm
I20260927 21:43:26.776440 17116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:26 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:26.779739 17116 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:43:26.781383 17116 webserver.cc:460] Webserver started at http://127.0.202.130:44131/ using document root <none> and password file <none>
I20260927 21:43:26.782049 17116 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:26.782160 17116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:26.787544 17116 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 21:43:26.790508 17131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:26.792050 17116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:43:26.792227 17116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "697123d814da43748e51e516cff01a47"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "317172484166d947cb486935f5edc643"
server_key_iv: "70a69ae21d0b3ebb118955785ebfd399"
server_key_version: "encryptionkey@0"
I20260927 21:43:26.793320 17116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:26.814113 17116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:26.818770 17116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:26.819067 17116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:26.820076 17116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:26.821493 17116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:26.821633 17116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:26.821805 17116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:26.821893 17116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:26.857088 17116 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:39303
I20260927 21:43:26.857136 17243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:39303 every 8 connection(s)
I20260927 21:43:26.858475 17116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:43:26.863864 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 17116
I20260927 21:43:26.864573 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:37005
--local_ip_for_outbound_sockets=127.0.202.131
--tserver_master_addrs=127.0.202.190:40549
--webserver_port=46825
--webserver_interface=127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:43:26 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:43:26.878813 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:26.861167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:50007 (local address 127.0.202.190:40549)
0927 21:43:26.861346 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:26.861351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:26.862532 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 21:43:26.864421 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:26.864426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:26.864538 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:26.864650 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:26.866152 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.866861 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.868441 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.868887 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.869031 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:26.872803 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:26.872808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:26.872812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:26.872846 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:26.875457 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:26.876243 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:26.876251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:26.876259 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:26.876422 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:26.876856 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:26.876861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:26.876865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:26.877316 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:26.877473 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:26.878192 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 21:43:26.878454 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:43:26.879863 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:26.861668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:50007)
0927 21:43:26.862371 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:26.862413 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:26.864123 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:26.864896 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:26.864904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:26.865431 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:43:26.865981 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:26.865999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.867031 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.867036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:26.868254 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:26.868266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:26.869128 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:26.869134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:26.869366 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:26.870084 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:43:26.870111 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:43:26.872537 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:43:26.875646 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:26.875656 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:26.875674 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:26.876103 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:26.876533 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:26.876538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:26.876541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:26.876737 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:26.877446 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:43:26.877454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:43:26.877811 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 21:43:26.878081 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 21:43:26.878717 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":276,"threads_started":1}
I20260927 21:43:26.881546 17244 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40549
I20260927 21:43:26.881876 17244 heartbeater.cc:464] Registering TS with master...
I20260927 21:43:26.882606 17244 heartbeater.cc:511] Master 127.0.202.190:40549 requested a full tablet report, sending...
I20260927 21:43:26.884208 16926 ts_manager.cc:194] Registered new tserver with Master: 697123d814da43748e51e516cff01a47 (127.0.202.130:39303)
I20260927 21:43:26.885627 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:50007
W20260927 21:43:27.204336 17247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:27.204633 17247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:27.204728 17247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:27.212318 17247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:27.212440 17247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:27.212559 17247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:43:27.222507 17247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:37005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=46825
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:27.223779 17247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:27.225044 17247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:27.239118 17252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:27.239066 17255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:27.239115 17253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:27.240269 17247 server_base.cc:1034] running on GCE node
I20260927 21:43:27.240954 17247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:27.242118 17247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:27.243348 17247 hybrid_clock.cc:648] HybridClock initialized: now 1790545407243272 us; error 68 us; skew 500 ppm
I20260927 21:43:27.243703 17247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:27 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:27.247063 17247 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:43:27.248778 17247 webserver.cc:460] Webserver started at http://127.0.202.131:46825/ using document root <none> and password file <none>
I20260927 21:43:27.249464 17247 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:27.249579 17247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:27.255302 17247 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 21:43:27.258155 17262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:27.259747 17247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:43:27.259927 17247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "b1a488699b9645a89e834a24d3c60b1d"
format_stamp: "Formatted at 2026-09-27 21:43:04 on dist-test-slave-25dq"
server_key: "665357b12d6bd037c2288c6b4829db9a"
server_key_iv: "5f3288b96360139aef73ff1a7610aa23"
server_key_version: "encryptionkey@0"
I20260927 21:43:27.261023 17247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:27.276607 17247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:27.281375 17247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:27.281652 17247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:27.282629 17247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:27.284147 17247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:27.284281 17247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:27.284459 17247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:27.284549 17247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:27.319674 17247 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37005
I20260927 21:43:27.319764 17374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37005 every 8 connection(s)
I20260927 21:43:27.321233 17247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:43:27.329428 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 17247
Sep 27 21:43:27 dist-test-slave-25dq krb5kdc[16341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:43:27.343662 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.324023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:53417 (local address 127.0.202.190:40549)
0927 21:43:27.324224 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.324228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.325484 (+ 1256us) server_negotiation.cc:407] Connection header received
0927 21:43:27.327393 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.327398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.327491 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.327615 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:27.329357 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.330187 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.332220 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.332728 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.332865 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.336796 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:27.336803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:27.336809 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:27.336851 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:27.340065 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.340916 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.340927 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.340929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.341059 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.341538 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.341543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.341552 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.341902 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:27.342173 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.343060 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.343374 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:43:27.344858 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.324569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:53417)
0927 21:43:27.325312 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:27.325356 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:27.327114 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:27.327858 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:27.327866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:27.328594 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:43:27.329180 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.329197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.330344 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.330350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.332074 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.332086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.332960 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.332969 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.333315 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.334014 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:43:27.334042 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:43:27.336560 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:43:27.340217 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:27.340226 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:27.340239 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:27.340777 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:27.341209 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:43:27.341213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:43:27.341216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:43:27.341411 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:43:27.342069 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:43:27.342077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:43:27.342576 (+ 499us) client_negotiation.cc:769] Sending connection context
0927 21:43:27.342919 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 21:43:27.343461 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":256,"threads_started":1}
I20260927 21:43:27.345256 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.330349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.0.202.190:40549)
0927 21:43:27.330864 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.330871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.331249 (+ 378us) server_negotiation.cc:407] Connection header received
0927 21:43:27.331381 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.331387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.331454 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.331563 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:27.333110 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.334136 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.335488 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.335993 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.336094 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.338989 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:27.338995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:27.339000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:27.339049 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:27.342217 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.343080 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.343085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.343088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.343212 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.343702 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.343709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.343715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.344036 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:27.344344 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.344651 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.344884 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":259,"threads_started":1}
I20260927 21:43:27.346706 17375 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40549
I20260927 21:43:27.347100 17375 heartbeater.cc:464] Registering TS with master...
I20260927 21:43:27.348016 17375 heartbeater.cc:511] Master 127.0.202.190:40549 requested a full tablet report, sending...
I20260927 21:43:27.349566 16925 ts_manager.cc:194] Registered new tserver with Master: b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005)
I20260927 21:43:27.350468 16925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:53417
I20260927 21:43:27.360671 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:43:27.367388 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.361619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.202.190:40549)
0927 21:43:27.361800 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.361805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.362361 (+ 556us) server_negotiation.cc:407] Connection header received
0927 21:43:27.362638 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.362642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.362801 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.362933 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:43:27.363351 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.364079 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.365634 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.366063 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.366161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.366646 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:43:27.366866 (+ 220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 21:43:27.367145 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45030: token expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 21:43:27.367528 17378 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45030: token expired
W20260927 21:43:27.368059 16849 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.202.190:40549)
I20260927 21:43:27.381414 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.368651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.202.190:40549)
0927 21:43:27.368818 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.368821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.368836 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:43:27.368872 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.368875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.368929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.369007 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:27.370544 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.371282 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.372773 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.373174 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.373274 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.375997 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:27.376002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:27.376005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:27.376038 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:27.378668 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.379353 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.379357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.379359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.379509 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.379819 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.379822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.379824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.380202 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:27.380537 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.380859 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.381092 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:43:27.384948 16925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:43:27.392405 16925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:43:27.424589 17112 heartbeater.cc:503] Master 127.0.202.190:40549 was elected leader, sending a full tablet report...
I20260927 21:43:27.433320 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.404535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:45275 (local address 127.0.0.1:58416)
0927 21:43:27.420393 (+ 15858us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:27.420419 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:27.420603 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:27.425144 (+ 4541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:27.425152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:27.425197 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:27.425677 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.425695 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.427760 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.427769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.430338 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.430360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.432150 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.432157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.432419 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.432479 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:43:27.432601 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 21:43:27.432789 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15664,"mutex_wait_us":1,"thread_start_us":21406,"threads_started":1}
I20260927 21:43:27.433846 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.405165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.0.202.129:45275)
0927 21:43:27.419792 (+ 14627us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.419807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.420987 (+ 1180us) server_negotiation.cc:407] Connection header received
0927 21:43:27.421133 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.421145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.424569 (+ 3424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.424815 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:27.425905 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.427597 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.430516 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.431899 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.432047 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.433267 (+ 1220us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.433341 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.433499 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14362,"spinlock_wait_cycles":35072,"thread_start_us":14187,"threads_started":1}
I20260927 21:43:27.434330 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.406770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39303 (local address 127.0.0.1:41116)
0927 21:43:27.424781 (+ 18011us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:27.424799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:27.424928 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:27.426160 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:27.426165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:27.426194 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:27.426568 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.426579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.428433 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.428438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.430907 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.430921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.433136 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.433143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.433394 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.433607 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 21:43:27.433710 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:43:27.433855 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17829,"mutex_wait_us":111,"thread_start_us":13931,"threads_started":1}
I20260927 21:43:27.435057 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.407014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.0.202.130:39303)
0927 21:43:27.422945 (+ 15931us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.422958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.425066 (+ 2108us) server_negotiation.cc:407] Connection header received
0927 21:43:27.425176 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.425183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.425474 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.425739 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:27.426764 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.428251 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.431083 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.432633 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.432773 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.432956 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.434652 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.434805 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15688,"thread_start_us":15949,"threads_started":1}
I20260927 21:43:27.436199 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.404429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:37005 (local address 127.0.0.1:60558)
0927 21:43:27.405268 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:27.405323 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:27.405514 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:27.428944 (+ 23430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:27.428949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:27.428975 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:27.429446 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.429457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.431511 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.431518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.433584 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.433598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.435382 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.435387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.435553 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.435584 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:43:27.435656 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:43:27.435785 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":57,"thread_start_us":1456,"threads_started":1}
I20260927 21:43:27.437220 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.404718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.202.131:37005)
0927 21:43:27.428251 (+ 23533us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.428261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.428322 (+ 61us) server_negotiation.cc:407] Connection header received
0927 21:43:27.428433 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.428441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.428720 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.429667 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:27.429929 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.431383 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.433753 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.435145 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.436594 (+ 1449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.436803 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.436854 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.436964 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23346,"thread_start_us":23193,"threads_started":1}
I20260927 21:43:27.438107 17047 tablet_service.cc:1467] Processing CreateTablet for tablet f0248c9a128549cc8585e61d18b9e343 (DEFAULT_TABLE table=test-table [id=ab040aa16812414b91fb2e63cbb5a230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:43:27.438460 17179 tablet_service.cc:1467] Processing CreateTablet for tablet f0248c9a128549cc8585e61d18b9e343 (DEFAULT_TABLE table=test-table [id=ab040aa16812414b91fb2e63cbb5a230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:43:27.439775 17179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0248c9a128549cc8585e61d18b9e343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:43:27.439795 17047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0248c9a128549cc8585e61d18b9e343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:43:27.440208 17310 tablet_service.cc:1467] Processing CreateTablet for tablet f0248c9a128549cc8585e61d18b9e343 (DEFAULT_TABLE table=test-table [id=ab040aa16812414b91fb2e63cbb5a230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:43:27.441361 17310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0248c9a128549cc8585e61d18b9e343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:43:27.453195 17391 tablet_bootstrap.cc:492] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Bootstrap starting.
I20260927 21:43:27.455426 17392 tablet_bootstrap.cc:492] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Bootstrap starting.
I20260927 21:43:27.456216 17393 tablet_bootstrap.cc:492] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Bootstrap starting.
I20260927 21:43:27.458540 17391 tablet_bootstrap.cc:654] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Neither blocks nor log segments found. Creating new log.
I20260927 21:43:27.459440 17392 tablet_bootstrap.cc:654] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Neither blocks nor log segments found. Creating new log.
I20260927 21:43:27.459767 17393 tablet_bootstrap.cc:654] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Neither blocks nor log segments found. Creating new log.
I20260927 21:43:27.459887 17391 log.cc:824] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:27.460660 17392 log.cc:824] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:27.461048 17393 log.cc:824] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:27.463204 17391 tablet_bootstrap.cc:492] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: No bootstrap required, opened a new log
I20260927 21:43:27.463485 17391 ts_tablet_manager.cc:1397] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 21:43:27.464012 17393 tablet_bootstrap.cc:492] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: No bootstrap required, opened a new log
I20260927 21:43:27.464139 17392 tablet_bootstrap.cc:492] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: No bootstrap required, opened a new log
I20260927 21:43:27.464269 17393 ts_tablet_manager.cc:1397] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 21:43:27.464457 17392 ts_tablet_manager.cc:1397] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260927 21:43:27.472167 17392 raft_consensus.cc:348] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.472656 17392 raft_consensus.cc:374] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:43:27.472765 17392 raft_consensus.cc:729] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a75efd563904395b97d09f0350077f2, State: Initialized, Role: FOLLOWER
I20260927 21:43:27.473412 17392 consensus_queue.cc:260] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.474997 17391 raft_consensus.cc:348] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.474963 17393 raft_consensus.cc:348] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.475435 17391 raft_consensus.cc:374] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:43:27.475459 17393 raft_consensus.cc:374] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:43:27.475559 17393 raft_consensus.cc:729] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697123d814da43748e51e516cff01a47, State: Initialized, Role: FOLLOWER
I20260927 21:43:27.475628 17391 raft_consensus.cc:729] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a488699b9645a89e834a24d3c60b1d, State: Initialized, Role: FOLLOWER
I20260927 21:43:27.476382 17391 consensus_queue.cc:260] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.476382 17393 consensus_queue.cc:260] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.477845 17392 ts_tablet_manager.cc:1428] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 21:43:27.478576 17397 raft_consensus.cc:482] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:43:27.478803 17397 raft_consensus.cc:504] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.480717 17397 leader_election.cc:288] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005), 697123d814da43748e51e516cff01a47 (127.0.202.130:39303)
I20260927 21:43:27.486649 17375 heartbeater.cc:503] Master 127.0.202.190:40549 was elected leader, sending a full tablet report...
I20260927 21:43:27.487103 17393 ts_tablet_manager.cc:1428] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Time spent starting tablet: real 0.023s user 0.008s sys 0.013s
I20260927 21:43:27.487216 17391 ts_tablet_manager.cc:1428] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Time spent starting tablet: real 0.024s user 0.021s sys 0.000s
I20260927 21:43:27.487918 17244 heartbeater.cc:503] Master 127.0.202.190:40549 was elected leader, sending a full tablet report...
I20260927 21:43:27.492477 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.481802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:33309 (local address 127.0.202.131:37005)
0927 21:43:27.482054 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.482063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.483266 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 21:43:27.484374 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.484379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.484455 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.484550 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:27.486180 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.487352 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.490022 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.491292 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.491452 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.491506 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.492051 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.492189 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:43:27.492554 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.481379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:37005 (local address 127.0.202.129:33309)
0927 21:43:27.483039 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:27.483070 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:27.484134 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:27.484793 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:27.484799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:27.484829 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:27.485983 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.485997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.487574 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.487579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.489846 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.489861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.491696 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.491702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.491897 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.491934 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:43:27.492037 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:43:27.492179 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"thread_start_us":1236,"threads_started":1}
I20260927 21:43:27.493443 17330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0248c9a128549cc8585e61d18b9e343" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a488699b9645a89e834a24d3c60b1d" is_pre_election: true
I20260927 21:43:27.493966 17330 raft_consensus.cc:2454] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 0.
I20260927 21:43:27.494379 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.483312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:40807 (local address 127.0.202.130:39303)
0927 21:43:27.483557 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.483566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.485795 (+ 2229us) server_negotiation.cc:407] Connection header received
0927 21:43:27.487570 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.487576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.487653 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.487736 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:43:27.488833 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.489807 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.492397 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.493463 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.493644 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.493728 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.494008 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.494135 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:43:27.494446 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.483050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:39303 (local address 127.0.202.129:40807)
0927 21:43:27.485685 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:27.485703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:43:27.487385 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:43:27.488074 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:43:27.488078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:43:27.488110 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:43:27.488624 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.488637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.490041 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.490048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.492167 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:43:27.492182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.493673 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.493680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:43:27.493843 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.493865 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:43:27.493969 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:43:27.494103 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2498,"thread_start_us":2261,"threads_started":1}
I20260927 21:43:27.494681 17003 leader_election.cc:302] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a75efd563904395b97d09f0350077f2, b1a488699b9645a89e834a24d3c60b1d; no voters:
I20260927 21:43:27.495035 17397 raft_consensus.cc:2781] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:43:27.495196 17397 raft_consensus.cc:482] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:43:27.495306 17397 raft_consensus.cc:3037] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:43:27.495800 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0248c9a128549cc8585e61d18b9e343" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697123d814da43748e51e516cff01a47" is_pre_election: true
I20260927 21:43:27.496249 17199 raft_consensus.cc:2454] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 0.
I20260927 21:43:27.498150 17397 raft_consensus.cc:504] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.498776 17397 leader_election.cc:288] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 election: Requested vote from peers b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005), 697123d814da43748e51e516cff01a47 (127.0.202.130:39303)
I20260927 21:43:27.499254 17330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0248c9a128549cc8585e61d18b9e343" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a488699b9645a89e834a24d3c60b1d"
I20260927 21:43:27.499284 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0248c9a128549cc8585e61d18b9e343" candidate_uuid: "5a75efd563904395b97d09f0350077f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697123d814da43748e51e516cff01a47"
I20260927 21:43:27.499464 17330 raft_consensus.cc:3037] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:43:27.499472 17199 raft_consensus.cc:3037] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:43:27.501785 17199 raft_consensus.cc:2454] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 1.
I20260927 21:43:27.501787 17330 raft_consensus.cc:2454] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a75efd563904395b97d09f0350077f2 in term 1.
I20260927 21:43:27.502219 17003 leader_election.cc:302] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a75efd563904395b97d09f0350077f2, 697123d814da43748e51e516cff01a47; no voters:
I20260927 21:43:27.502543 17397 raft_consensus.cc:2781] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:43:27.503055 17397 raft_consensus.cc:686] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5a75efd563904395b97d09f0350077f2, State: Running, Role: LEADER
I20260927 21:43:27.503682 17397 consensus_queue.cc:237] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } }
I20260927 21:43:27.508476 16925 catalog_manager.cc:5495] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a75efd563904395b97d09f0350077f2 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "5a75efd563904395b97d09f0350077f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a75efd563904395b97d09f0350077f2" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 45275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 } health_report { overall_health: UNKNOWN } } }
I20260927 21:43:27.542128 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.536429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.0.202.129:45275)
0927 21:43:27.536690 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.536695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.536714 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:43:27.536895 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.536898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.537004 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.537154 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:43:27.537558 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.538369 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.539802 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.540255 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.540443 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.540914 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:43:27.541493 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:43:27.541611 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.541805 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.541884 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 21:43:27.551132 17330 raft_consensus.cc:1261] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Refusing update from remote peer 5a75efd563904395b97d09f0350077f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:43:27.551328 17199 raft_consensus.cc:1261] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Refusing update from remote peer 5a75efd563904395b97d09f0350077f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:43:27.552326 17402 consensus_queue.cc:1035] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a488699b9645a89e834a24d3c60b1d" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:43:27.552724 17397 consensus_queue.cc:1035] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "697123d814da43748e51e516cff01a47" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:43:27.564782 17404 mvcc.cc:205] Tried to move back new op lower bound from 7334073989316022272 to 7334073989137178624. Current Snapshot: MvccSnapshot[applied={T|T < 7334073989316022272}]
I20260927 21:43:27.566004 17406 mvcc.cc:205] Tried to move back new op lower bound from 7334073989316022272 to 7334073989137178624. Current Snapshot: MvccSnapshot[applied={T|T < 7334073989316022272}]
I20260927 21:43:27.567524 17405 mvcc.cc:205] Tried to move back new op lower bound from 7334073989316022272 to 7334073989137178624. Current Snapshot: MvccSnapshot[applied={T|T < 7334073989316022272}]
I20260927 21:43:27.580101 16926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:43:27.580406 16926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:43:27.584352 16926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Sending DeleteTablet for 3 replicas of tablet f0248c9a128549cc8585e61d18b9e343
I20260927 21:43:27.585671 17047 tablet_service.cc:1514] Processing DeleteTablet for tablet f0248c9a128549cc8585e61d18b9e343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:43:27 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:58416
I20260927 21:43:27.585911 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16963
I20260927 21:43:27.586035 17310 tablet_service.cc:1514] Processing DeleteTablet for tablet f0248c9a128549cc8585e61d18b9e343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:43:27 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:60558
I20260927 21:43:27.587281 17179 tablet_service.cc:1514] Processing DeleteTablet for tablet f0248c9a128549cc8585e61d18b9e343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:43:27 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:41116
I20260927 21:43:27.597160 17414 tablet_replica.cc:330] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: stopping tablet replica
I20260927 21:43:27.597345 17416 tablet_replica.cc:330] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: stopping tablet replica
I20260927 21:43:27.597676 17414 raft_consensus.cc:2229] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:43:27.598312 17416 raft_consensus.cc:2229] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:43:27.598378 17414 raft_consensus.cc:2258] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:43:27.598913 17416 raft_consensus.cc:2258] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:43:27.600771 17414 ts_tablet_manager.cc:1905] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:43:27.602363 17416 ts_tablet_manager.cc:1905] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:43:27.607292 17415 tablet_replica.cc:330] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: stopping tablet replica
I20260927 21:43:27.607788 17415 raft_consensus.cc:2229] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:43:27.608424 17415 raft_consensus.cc:2258] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:43:27.608958 17414 ts_tablet_manager.cc:1918] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:43:27.609159 17414 log.cc:1196] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/f0248c9a128549cc8585e61d18b9e343
I20260927 21:43:27.609722 17414 ts_tablet_manager.cc:1939] T f0248c9a128549cc8585e61d18b9e343 P 5a75efd563904395b97d09f0350077f2: Deleting consensus metadata
I20260927 21:43:27.609961 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.586845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.0.202.129:45275)
0927 21:43:27.587590 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.587599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.587621 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:43:27.587688 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.587696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.587816 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.587928 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:27.590335 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.591893 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.594498 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.595490 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.595637 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.600099 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:27.600119 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:27.600142 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:27.600210 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:27.604773 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.606087 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.606104 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.606110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.606396 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.606963 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.606968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.606971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.607466 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:27.607759 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.608500 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.608808 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589}
I20260927 21:43:27.610690 17107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:27.612859 17415 ts_tablet_manager.cc:1905] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:43:27.615269 17416 ts_tablet_manager.cc:1918] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:43:27.615453 17416 log.cc:1196] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/f0248c9a128549cc8585e61d18b9e343
I20260927 21:43:27.615924 17416 ts_tablet_manager.cc:1939] T f0248c9a128549cc8585e61d18b9e343 P 697123d814da43748e51e516cff01a47: Deleting consensus metadata
I20260927 21:43:27.617599 16913 catalog_manager.cc:4841] TS 697123d814da43748e51e516cff01a47 (127.0.202.130:39303): tablet f0248c9a128549cc8585e61d18b9e343 (table test-table [id=ab040aa16812414b91fb2e63cbb5a230]) successfully deleted
I20260927 21:43:27.620685 17415 ts_tablet_manager.cc:1918] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:43:27.620858 17415 log.cc:1196] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/f0248c9a128549cc8585e61d18b9e343
I20260927 21:43:27.621397 17415 ts_tablet_manager.cc:1939] T f0248c9a128549cc8585e61d18b9e343 P b1a488699b9645a89e834a24d3c60b1d: Deleting consensus metadata
I20260927 21:43:27.623409 16913 catalog_manager.cc:4841] TS b1a488699b9645a89e834a24d3c60b1d (127.0.202.131:37005): tablet f0248c9a128549cc8585e61d18b9e343 (table test-table [id=ab040aa16812414b91fb2e63cbb5a230]) successfully deleted
I20260927 21:43:27.713207 16910 catalog_manager.cc:4841] TS 5a75efd563904395b97d09f0350077f2 (127.0.202.129:45275): tablet f0248c9a128549cc8585e61d18b9e343 (table test-table [id=ab040aa16812414b91fb2e63cbb5a230]) successfully deleted
I20260927 21:43:27.717000 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16963
I20260927 21:43:27.735455 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 17116
I20260927 21:43:27.750301 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.736238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.0.202.130:39303)
0927 21:43:27.736555 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.736560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.736577 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:43:27.736634 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.736638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.736705 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.736789 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:27.738207 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.739049 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.740472 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.740869 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.740970 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.743797 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:27.743814 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:27.743830 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:27.743870 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:27.746974 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.747930 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.747938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.747940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.748098 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.748635 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.748639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.748641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.749040 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:27.749280 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.749646 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.749843 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260927 21:43:27.751017 17239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:27.841163 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 17116
I20260927 21:43:27.863994 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 17247
I20260927 21:43:27.880412 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:27.864859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.0.202.131:37005)
0927 21:43:27.865044 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:43:27.865048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:43:27.865132 (+ 84us) server_negotiation.cc:407] Connection header received
0927 21:43:27.865333 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:43:27.865337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:43:27.865402 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:43:27.865486 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:43:27.867327 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.868073 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.870127 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:43:27.870553 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:43:27.870661 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:43:27.874109 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:43:27.874125 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:43:27.874141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:43:27.874183 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:43:27.877306 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.878119 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.878127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.878129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.878331 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:43:27.878742 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:43:27.878748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:43:27.878751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:43:27.879119 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:43:27.879421 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 21:43:27.879764 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 21:43:27.879991 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:43:27.881072 17370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:27.971614 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 17247
I20260927 21:43:27.989601 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 16893
I20260927 21:43:27.990876 16956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:43:28.107270 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 16893
I20260927 21:43:48.129618 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40549
--webserver_interface=127.0.202.190
--webserver_port=37579
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:48.469033 17421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:48.469326 17421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:48.469415 17421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:48.476696 17421 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:43:48.476802 17421 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:43:48.476866 17421 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:43:48.476944 17421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:43:48.477011 17421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:48.477075 17421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:48.477120 17421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:43:48.477167 17421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:43:48.487147 17421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=37579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:48.488423 17421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:48.489850 17421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:48.502812 17427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:48.502873 17429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:48.502876 17426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:48.504086 17421 server_base.cc:1034] running on GCE node
I20260927 21:43:48.505002 17421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:48.506515 17421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:48.507737 17421 hybrid_clock.cc:648] HybridClock initialized: now 1790545428507655 us; error 59 us; skew 500 ppm
I20260927 21:43:48.508060 17421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:48 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:48.517257 17421 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:43:48.518998 17421 webserver.cc:460] Webserver started at http://127.0.202.190:37579/ using document root <none> and password file <none>
I20260927 21:43:48.519786 17421 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:48.519930 17421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:48.525883 17421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 21:43:48.529193 17436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:48.530822 17421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:43:48.531025 17421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ad7e6520141442caa1747bb8d02a2aee"
format_stamp: "Formatted at 2026-09-27 21:43:02 on dist-test-slave-25dq"
server_key: "b7bdf5b0bb2df3c7544c1c99b2dcf6cc"
server_key_iv: "1df39b67cc836ba213293c287b2e7891"
server_key_version: "encryptionkey@0"
I20260927 21:43:48.532043 17421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:48.547338 17421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:48.551714 17421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:48.552023 17421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:48.571763 17421 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40549
I20260927 21:43:48.571786 17487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40549 every 8 connection(s)
I20260927 21:43:48.573403 17421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:43:48.575361 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 17421
I20260927 21:43:48.576212 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:45275
--local_ip_for_outbound_sockets=127.0.202.129
--tserver_master_addrs=127.0.202.190:40549
--webserver_port=40197
--webserver_interface=127.0.202.129
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:43:48.581326 17488 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:43:48.585932 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Bootstrap starting.
I20260927 21:43:48.617823 17488 log.cc:824] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Log is configured to *not* fsync() on all Append() calls
I20260927 21:43:48.639122 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:43:48.639816 17488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Bootstrap complete.
I20260927 21:43:48.654887 17488 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:48.656206 17488 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7e6520141442caa1747bb8d02a2aee, State: Initialized, Role: FOLLOWER
I20260927 21:43:48.656980 17488 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:48.657269 17488 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:43:48.657395 17488 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:43:48.657572 17488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:43:48.663157 17488 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:48.663728 17488 leader_election.cc:302] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad7e6520141442caa1747bb8d02a2aee; no voters:
I20260927 21:43:48.664496 17488 leader_election.cc:288] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:43:48.664657 17492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:43:48.666078 17492 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [term 3 LEADER]: Becoming Leader. State: Replica: ad7e6520141442caa1747bb8d02a2aee, State: Running, Role: LEADER
I20260927 21:43:48.666589 17488 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:43:48.666702 17492 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } }
I20260927 21:43:48.669430 17493 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ad7e6520141442caa1747bb8d02a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } } }
I20260927 21:43:48.669391 17494 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad7e6520141442caa1747bb8d02a2aee. Latest consensus state: current_term: 3 leader_uuid: "ad7e6520141442caa1747bb8d02a2aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7e6520141442caa1747bb8d02a2aee" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40549 } } }
I20260927 21:43:48.669831 17494 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: This master's current role is: LEADER
I20260927 21:43:48.669829 17493 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee [sys.catalog]: This master's current role is: LEADER
I20260927 21:43:48.672081 17499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:43:48.680718 17499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:43:48.682241 17499 catalog_manager.cc:1259] Loaded cluster ID: 4d5074f665474e6fa8b8434686f0445a
I20260927 21:43:48.682353 17499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:43:48.686582 17499 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:43:48.688616 17499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Loaded TSK: 0
I20260927 21:43:48.688767 17499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad7e6520141442caa1747bb8d02a2aee: Loaded TSK: 1
F20260927 21:43:48.690171 17499 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7b58df49ad google::LogMessage::Fail() at ??:0
@ 0x7f7b58df788a google::LogMessage::SendToLog() at ??:0
@ 0x7f7b58df44de google::LogMessage::Flush() at ??:0
@ 0x7f7b58df5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7b5cb587d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7b5cb4773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7b6e8231c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7b6e822783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7b6e8ae2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7b6e8ae0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7b6e8f85e9 std::function<>::operator()() at ??:0
@ 0x7f7b6e826cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7b6e824c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7b6e8ac610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7b6e8ac439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7b5ba40345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7b5ba4391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7b5ba43799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7b5ba1eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7b5ecf86db start_thread at ??:0
@ 0x7f7b57c8e61f clone at ??:0
W20260927 21:43:49.018993 17490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:49.019330 17490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:49.019507 17490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:49.027920 17490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:49.028049 17490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:49.028223 17490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:43:49.038349 17490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:45275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=40197
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:49.039747 17490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:49.041174 17490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:49.055425 17535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:49.056233 17538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:49.056187 17490 server_base.cc:1034] running on GCE node
W20260927 21:43:49.056280 17536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:49.057039 17490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:49.058182 17490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:49.059389 17490 hybrid_clock.cc:648] HybridClock initialized: now 1790545429059335 us; error 41 us; skew 500 ppm
I20260927 21:43:49.059746 17490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:49 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:49.062915 17490 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:43:49.064435 17490 webserver.cc:460] Webserver started at http://127.0.202.129:40197/ using document root <none> and password file <none>
I20260927 21:43:49.065104 17490 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:49.065207 17490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:49.070767 17490 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:43:49.073722 17545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:49.075212 17490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:43:49.075433 17490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "5a75efd563904395b97d09f0350077f2"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "9107b4191cfddbd01da7b31bfd1da282"
server_key_iv: "393dd8f0958f72182da4436f602c1ed1"
server_key_version: "encryptionkey@0"
I20260927 21:43:49.076541 17490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:49.095988 17490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:49.100806 17490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:49.101127 17490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:49.102128 17490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:49.103577 17490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:49.103724 17490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:49.103863 17490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:49.103951 17490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:49.139552 17490 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:45275
I20260927 21:43:49.139585 17657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:45275 every 8 connection(s)
I20260927 21:43:49.141185 17490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:43:49.146387 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:49.144438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:33415)
0927 21:43:49.145195 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:49.145433 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":291,"threads_started":1}
W20260927 21:43:49.147221 17658 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:40549 (0 consecutive failures): Network error: Failed to ping master at 127.0.202.190:40549: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
I20260927 21:43:49.148041 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:49.147717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:58255)
0927 21:43:49.147856 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:49.147936 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 21:43:49.148720 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 17490
I20260927 21:43:49.148999 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:49.148665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:59165)
0927 21:43:49.148807 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:49.148895 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:43:49.149286 17658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:43:49.149504 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:39303
--local_ip_for_outbound_sockets=127.0.202.130
--tserver_master_addrs=127.0.202.190:40549
--webserver_port=44131
--webserver_interface=127.0.202.130
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:49.486718 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:49.487062 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:49.487165 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:49.494598 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:49.494787 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:49.495003 17661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:43:49.505151 17661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:39303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=44131
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:49.506472 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:49.507750 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:49.521618 17666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:49.521667 17667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:49.521845 17669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:49.522513 17661 server_base.cc:1034] running on GCE node
I20260927 21:43:49.523151 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:49.524082 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:49.525312 17661 hybrid_clock.cc:648] HybridClock initialized: now 1790545429525234 us; error 51 us; skew 500 ppm
I20260927 21:43:49.525619 17661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:49 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:49.528829 17661 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:43:49.530483 17661 webserver.cc:460] Webserver started at http://127.0.202.130:44131/ using document root <none> and password file <none>
I20260927 21:43:49.531168 17661 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:49.531271 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:49.536791 17661 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:43:49.539701 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:49.541308 17661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:43:49.541498 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "697123d814da43748e51e516cff01a47"
format_stamp: "Formatted at 2026-09-27 21:43:03 on dist-test-slave-25dq"
server_key: "317172484166d947cb486935f5edc643"
server_key_iv: "70a69ae21d0b3ebb118955785ebfd399"
server_key_version: "encryptionkey@0"
I20260927 21:43:49.542505 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:49.572376 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:49.577095 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:49.577371 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:49.578433 17661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:49.580247 17661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:49.580392 17661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 21:43:49.580533 17661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:49.580626 17661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:49.615594 17661 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:39303
I20260927 21:43:49.615687 17788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:39303 every 8 connection(s)
I20260927 21:43:49.617120 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:43:49.622366 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:49.620506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:35119)
0927 21:43:49.621267 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:49.621508 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
W20260927 21:43:49.623034 17789 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:40549 (0 consecutive failures): Network error: Failed to ping master at 127.0.202.190:40549: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
I20260927 21:43:49.623912 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:49.623580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:49057)
0927 21:43:49.623736 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:49.623809 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 21:43:49.624660 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 17661
I20260927 21:43:49.624927 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:49.624540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:37545)
0927 21:43:49.624700 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:49.624799 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 21:43:49.625218 17789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:43:49.625473 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:37005
--local_ip_for_outbound_sockets=127.0.202.131
--tserver_master_addrs=127.0.202.190:40549
--webserver_port=46825
--webserver_interface=127.0.202.131
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:43:49.974967 17792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:43:49.975291 17792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:43:49.975504 17792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:43:49.983590 17792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:43:49.983739 17792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:43:49.983987 17792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:43:49.994256 17792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:37005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=46825
--tserver_master_addrs=127.0.202.190:40549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:43:49.995710 17792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:43:49.997283 17792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:43:50.011843 17800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:50.011785 17798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:43:50.011817 17797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:43:50.014324 17792 server_base.cc:1034] running on GCE node
I20260927 21:43:50.015043 17792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:43:50.016201 17792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:43:50.017663 17792 hybrid_clock.cc:648] HybridClock initialized: now 1790545430017603 us; error 49 us; skew 500 ppm
I20260927 21:43:50.018107 17792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:43:50 dist-test-slave-25dq krb5kdc[16341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:43:50.021786 17792 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:43:50.023576 17792 webserver.cc:460] Webserver started at http://127.0.202.131:46825/ using document root <none> and password file <none>
I20260927 21:43:50.024394 17792 fs_manager.cc:359] Metadata directory not provided
I20260927 21:43:50.024508 17792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:43:50.030223 17792 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:43:50.033123 17807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:50.034535 17792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:43:50.034673 17792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "b1a488699b9645a89e834a24d3c60b1d"
format_stamp: "Formatted at 2026-09-27 21:43:04 on dist-test-slave-25dq"
server_key: "665357b12d6bd037c2288c6b4829db9a"
server_key_iv: "5f3288b96360139aef73ff1a7610aa23"
server_key_version: "encryptionkey@0"
I20260927 21:43:50.035610 17792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:43:50.070175 17792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:43:50.074344 17792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:43:50.074580 17792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:43:50.075598 17792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:43:50.077024 17792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:43:50.077111 17792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:50.077255 17792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:43:50.077333 17792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:43:50.113117 17792 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:37005
I20260927 21:43:50.113279 17919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:37005 every 8 connection(s)
I20260927 21:43:50.114615 17792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:43:50.119719 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:50.117761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:33577)
0927 21:43:50.118525 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:50.118822 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":311,"threads_started":1}
W20260927 21:43:50.120719 17920 heartbeater.cc:644] Failed to heartbeat to 127.0.202.190:40549 (0 consecutive failures): Network error: Failed to ping master at 127.0.202.190:40549: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
I20260927 21:43:50.120806 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 17792
I20260927 21:43:50.121624 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:50.121242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:47549)
0927 21:43:50.121422 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:50.121502 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 21:43:50.122686 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:50.122331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:53547)
0927 21:43:50.122463 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:50.122537 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:43:50.122979 17920 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:43:50.150744 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:50.149847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:50981)
0927 21:43:50.150362 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:50.150480 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":239,"threads_started":1}
I20260927 21:43:50.626902 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:50.625894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:49299)
0927 21:43:50.626486 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:50.626657 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":230,"threads_started":1}
I20260927 21:43:51.124559 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:51.123595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:37675)
0927 21:43:51.124120 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:51.124211 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":233,"threads_started":1}
I20260927 21:43:51.152842 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:51.151964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:51107)
0927 21:43:51.152585 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:51.152670 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":369,"threads_started":1}
I20260927 21:43:51.629192 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:51.628119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:47235)
0927 21:43:51.628786 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:51.628980 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":390,"threads_started":1}
I20260927 21:43:52.126713 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:52.125704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:36161)
0927 21:43:52.126404 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:52.126518 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":403,"threads_started":1}
I20260927 21:43:52.154690 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:52.153892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:46729)
0927 21:43:52.154402 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:52.154496 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
I20260927 21:43:52.631424 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:52.630410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:57741)
0927 21:43:52.631027 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:52.631149 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":319,"threads_started":1}
I20260927 21:43:53.128759 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:53.127761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:36121)
0927 21:43:53.128390 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:53.128478 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":348,"threads_started":1}
I20260927 21:43:53.156610 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:53.155751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:39637)
0927 21:43:53.156223 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:53.156390 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260927 21:43:53.633348 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:53.632608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:35727)
0927 21:43:53.633072 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:53.633172 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260927 21:43:54.130797 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:54.129884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:44607)
0927 21:43:54.130409 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:54.130526 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
I20260927 21:43:54.158589 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:54.157834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:56541)
0927 21:43:54.158324 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:54.158408 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
I20260927 21:43:54.635486 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:54.634474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:45941)
0927 21:43:54.635055 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:54.635211 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":299,"threads_started":1}
I20260927 21:43:55.132577 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:55.131788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:58747)
0927 21:43:55.132259 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:55.132402 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":263,"threads_started":1}
I20260927 21:43:55.160501 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:55.159608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:58769)
0927 21:43:55.160177 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:55.160267 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":357,"threads_started":1}
I20260927 21:43:55.637504 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:55.636641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:46393)
0927 21:43:55.637166 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:55.637274 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
I20260927 21:43:56.134440 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:56.133561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:41007)
0927 21:43:56.134123 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:56.134227 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":309,"threads_started":1}
I20260927 21:43:56.162513 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:56.161572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:49327)
0927 21:43:56.162184 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:56.162270 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":322,"threads_started":1}
I20260927 21:43:56.639539 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:56.638558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:56489)
0927 21:43:56.639143 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:56.639269 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":291,"threads_started":1}
I20260927 21:43:57.136234 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:57.135427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:47675)
0927 21:43:57.135917 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:57.136002 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":213,"threads_started":1}
I20260927 21:43:57.164443 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:57.163602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:44469)
0927 21:43:57.164088 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:57.164191 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":271,"threads_started":1}
I20260927 21:43:57.641542 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:57.640704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:50537)
0927 21:43:57.641223 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:57.641329 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":268,"threads_started":1}
I20260927 21:43:58.137984 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:58.137264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:48135)
0927 21:43:58.137720 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:58.137821 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260927 21:43:58.166247 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:58.165397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:46003)
0927 21:43:58.165975 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:58.166058 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":329,"threads_started":1}
I20260927 21:43:58.643505 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:58.642621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:34795)
0927 21:43:58.643217 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:58.643335 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":306,"threads_started":1}
I20260927 21:43:59.140053 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:59.139069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:55529)
0927 21:43:59.139672 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:59.139826 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":290,"threads_started":1}
I20260927 21:43:59.168092 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:59.167228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:37363)
0927 21:43:59.167828 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:59.167923 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260927 21:43:59.645840 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:43:59.644721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:57613)
0927 21:43:59.645439 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 21:43:59.645570 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":402,"threads_started":1}
I20260927 21:44:00.141824 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:00.141013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:33261)
0927 21:44:00.141531 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:00.141635 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":282,"threads_started":1}
I20260927 21:44:00.170132 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:00.169234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:47783)
0927 21:44:00.169744 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:00.169860 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
I20260927 21:44:00.648070 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:00.647052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:43163)
0927 21:44:00.647670 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:00.647792 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":266,"threads_started":1}
I20260927 21:44:01.143440 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:01.142712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:37625)
0927 21:44:01.143173 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:01.143269 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
I20260927 21:44:01.172261 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:01.171378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:33223)
0927 21:44:01.171961 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:01.172055 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":314,"threads_started":1}
I20260927 21:44:01.650084 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:01.649291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:50109)
0927 21:44:01.649816 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:01.649922 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260927 21:44:02.145043 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:02.144369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:40959)
0927 21:44:02.144811 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:02.144891 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":197,"threads_started":1}
I20260927 21:44:02.174551 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:02.173624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:58271)
0927 21:44:02.174207 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:02.174337 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":276,"threads_started":1}
I20260927 21:44:02.652101 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:02.651188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:47827)
0927 21:44:02.651785 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:02.651900 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":258,"threads_started":1}
I20260927 21:44:03.147042 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:03.146132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:53079)
0927 21:44:03.146762 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:03.146854 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":319,"threads_started":1}
I20260927 21:44:03.176676 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:03.175776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:34481)
0927 21:44:03.176373 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:03.176460 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":318,"threads_started":1}
I20260927 21:44:03.654289 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:03.653239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:39163)
0927 21:44:03.653901 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:03.654019 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":362,"threads_started":1}
I20260927 21:44:04.148986 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:04.148001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:48461)
0927 21:44:04.148603 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:04.148756 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":335,"threads_started":1}
I20260927 21:44:04.178802 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:04.177938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:50505)
0927 21:44:04.178476 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:04.178561 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":316,"threads_started":1}
I20260927 21:44:04.656567 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:04.655610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.130:34999)
0927 21:44:04.656189 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:04.656338 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":320,"threads_started":1}
I20260927 21:44:05.127841 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 17490
I20260927 21:44:05.147648 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:05.129220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.0.202.129:45275)
0927 21:44:05.129945 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:05.129960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:05.129999 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:44:05.131610 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:05.131620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:05.132070 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:05.132464 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:05.133840 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:05.135051 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:05.136656 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:05.137374 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:05.137512 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:05.140807 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:05.140816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:05.140833 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:05.140873 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:05.144129 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:05.144914 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:05.144921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:05.144928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:05.145110 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:05.145458 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:05.145461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:05.145463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:05.146169 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:05.146472 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:05.146834 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 21:44:05.147289 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":418,"threads_started":1}
I20260927 21:44:05.149207 17653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:05.151141 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:05.150146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.131:43681)
0927 21:44:05.150679 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:05.150836 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":282,"threads_started":1}
I20260927 21:44:05.237493 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:05.236266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40549 (local address 127.0.202.129:35595)
0927 21:44:05.236901 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:05.237152 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.190:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":233,"threads_started":1}
I20260927 21:44:05.237475 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 17490
I20260927 21:44:05.256163 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 17661
I20260927 21:44:05.274772 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:05.257224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.0.202.130:39303)
0927 21:44:05.257817 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:05.257831 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:05.257869 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:44:05.259450 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:05.259460 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:05.259915 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:05.260231 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:05.261853 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:05.263088 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:05.264374 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:05.265108 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:05.265254 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:05.268088 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:05.268095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:05.268112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:05.268149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:05.271422 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:05.272147 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:05.272153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:05.272161 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:05.272397 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:05.272838 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:05.272841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:05.272843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:05.273419 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:05.273654 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:05.274079 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 21:44:05.274426 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":292,"threads_started":1}
I20260927 21:44:05.276281 17784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:05.363440 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 17661
I20260927 21:44:05.381701 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 17792
I20260927 21:44:05.400345 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:05.382730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.202.131:37005)
0927 21:44:05.383333 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:05.383349 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:05.383390 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:44:05.385122 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:05.385133 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:05.385594 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:05.385900 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:05.387307 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:05.388741 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:05.390110 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:05.390799 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:05.390929 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:05.393806 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:05.393815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:05.393831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:05.393882 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:05.397082 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:05.397828 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:05.397835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:05.397842 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:05.397998 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:05.398321 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:05.398324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:05.398326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:05.398851 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:05.399073 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:05.399533 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 21:44:05.399957 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":273,"threads_started":1}
I20260927 21:44:05.401724 17915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:05.487954 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 17792
2026-09-27T21:44:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64636 ms)
[----------] 30 tests from SecurityITest (309646 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:05 dist-test-slave-25dq krb5kdc[17978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:05 dist-test-slave-25dq krb5kdc[17978](info): set up 2 sockets
Sep 27 21:44:05 dist-test-slave-25dq krb5kdc[17978](info): commencing operation
krb5kdc: starting...
W20260927 21:44:06.644565 810 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:06 dist-test-slave-25dq krb5kdc[17978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:44:06.918198 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44351
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:07.259045 17991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:07.259374 17991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:07.259526 17991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:07.267266 17991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:07.267408 17991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:07.267516 17991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:07.267627 17991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:07.267688 17991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:07.278070 17991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:07.279554 17991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:07.281034 17991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:07.294643 17996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:07.294672 17999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:07.294673 17997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:07.295323 17991 server_base.cc:1034] running on GCE node
I20260927 21:44:07.296258 17991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:07.298034 17991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:07.299268 17991 hybrid_clock.cc:648] HybridClock initialized: now 1790545447299198 us; error 61 us; skew 500 ppm
I20260927 21:44:07.299777 17991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:07 dist-test-slave-25dq krb5kdc[17978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:07.309218 17991 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:44:07.310897 17991 webserver.cc:460] Webserver started at http://127.0.202.190:35651/ using document root <none> and password file <none>
I20260927 21:44:07.311676 17991 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:07.311821 17991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:07.312178 17991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:07.315219 17991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "5db2b46f54d24cfb8e6c80d9f55e8469"
format_stamp: "Formatted at 2026-09-27 21:44:07 on dist-test-slave-25dq"
server_key: "55d6c84aa2de9a7e745bdc19ea613877"
server_key_iv: "71eb5f074e87e871366dcc7cd4bd1929"
server_key_version: "encryptionkey@0"
I20260927 21:44:07.315989 17991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "5db2b46f54d24cfb8e6c80d9f55e8469"
format_stamp: "Formatted at 2026-09-27 21:44:07 on dist-test-slave-25dq"
server_key: "55d6c84aa2de9a7e745bdc19ea613877"
server_key_iv: "71eb5f074e87e871366dcc7cd4bd1929"
server_key_version: "encryptionkey@0"
I20260927 21:44:07.321363 17991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:44:07.325073 18006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:07.326555 17991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:07.326766 17991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "5db2b46f54d24cfb8e6c80d9f55e8469"
format_stamp: "Formatted at 2026-09-27 21:44:07 on dist-test-slave-25dq"
server_key: "55d6c84aa2de9a7e745bdc19ea613877"
server_key_iv: "71eb5f074e87e871366dcc7cd4bd1929"
server_key_version: "encryptionkey@0"
I20260927 21:44:07.326979 17991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:07.339314 17991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:07.343899 17991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:07.344214 17991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:07.364190 17991 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:44351
I20260927 21:44:07.364215 18057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:44351 every 8 connection(s)
I20260927 21:44:07.365744 17991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:07.369753 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:07.374379 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 17991
I20260927 21:44:07.374598 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:07.375308 810 external_mini_cluster.cc:1428] Setting key 7ffce26088f4b0545e71f633c04b125d
I20260927 21:44:07.382741 18058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469: Bootstrap starting.
I20260927 21:44:07.386837 18058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:07.388067 18058 log.cc:824] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:07.391552 18058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469: No bootstrap required, opened a new log
Sep 27 21:44:07 dist-test-slave-25dq krb5kdc[17978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:07.403275 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:07.377514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.0.202.190:44351)
0927 21:44:07.378688 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:07.378704 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:07.378761 (+ 57us) server_negotiation.cc:407] Connection header received
0927 21:44:07.381273 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:07.381306 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:07.381854 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:07.382341 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:07.383837 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:07.386004 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:07.387741 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:07.388921 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:07.389074 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:07.393591 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:07.393608 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:07.393634 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:07.393701 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:07.398218 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:07.399056 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:07.399067 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:07.399077 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:07.399334 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:07.399713 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:07.399726 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:07.399729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:07.400561 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:07.400915 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:07.401236 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 21:44:07.401758 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":606,"threads_started":1}
I20260927 21:44:07.403391 18058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44351 } }
I20260927 21:44:07.403884 18058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:07.404031 18058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db2b46f54d24cfb8e6c80d9f55e8469, State: Initialized, Role: FOLLOWER
I20260927 21:44:07.404776 18058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44351 } }
I20260927 21:44:07.405069 18058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:07.405251 18058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:07.405469 18058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:07.407991 18058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44351 } }
I20260927 21:44:07.408676 18058 leader_election.cc:302] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5db2b46f54d24cfb8e6c80d9f55e8469; no voters:
I20260927 21:44:07.409336 18058 leader_election.cc:288] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:07.409469 18063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:07.411010 18063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [term 1 LEADER]: Becoming Leader. State: Replica: 5db2b46f54d24cfb8e6c80d9f55e8469, State: Running, Role: LEADER
I20260927 21:44:07.411597 18063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44351 } }
I20260927 21:44:07.412236 18058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:07.415158 18065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5db2b46f54d24cfb8e6c80d9f55e8469. Latest consensus state: current_term: 1 leader_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44351 } } }
I20260927 21:44:07.415635 18065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:07.415514 18064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db2b46f54d24cfb8e6c80d9f55e8469" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44351 } } }
I20260927 21:44:07.415915 18064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:07.417626 18072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:07.423099 18072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:07.433324 18072 catalog_manager.cc:1347] Generated new cluster ID: 8adc8434d63b439aab1131e79d0c15f0
I20260927 21:44:07.433511 18072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:07.453615 18072 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:07.455004 18072 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:07.462865 18072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469: Generated new TSK 0
I20260927 21:44:07.463721 18072 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:44:07.536330 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44351
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:07.888033 18084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:07.888417 18084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:07.888592 18084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:07.896245 18084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:07.896490 18084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:07.896684 18084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:44:07.907229 18084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:07.908720 18084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:07.910076 18084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:07.924175 18090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:07.924109 18092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:07.924245 18089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:07.925284 18084 server_base.cc:1034] running on GCE node
I20260927 21:44:07.926076 18084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:07.927035 18084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:07.928330 18084 hybrid_clock.cc:648] HybridClock initialized: now 1790545447928230 us; error 52 us; skew 500 ppm
I20260927 21:44:07.928755 18084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:07 dist-test-slave-25dq krb5kdc[17978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:07.932157 18084 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:44:07.933843 18084 webserver.cc:460] Webserver started at http://127.0.202.129:36379/ using document root <none> and password file <none>
I20260927 21:44:07.934557 18084 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:07.934706 18084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:07.935021 18084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:07.938028 18084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "8c1766a4849c4cb386a5ffede17df2c5"
format_stamp: "Formatted at 2026-09-27 21:44:07 on dist-test-slave-25dq"
server_key: "b2fc85f9c5546f0770cba5e883dc34fa"
server_key_iv: "c53ac858221e8ab46b34e3e52f830007"
server_key_version: "encryptionkey@0"
I20260927 21:44:07.938844 18084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "8c1766a4849c4cb386a5ffede17df2c5"
format_stamp: "Formatted at 2026-09-27 21:44:07 on dist-test-slave-25dq"
server_key: "b2fc85f9c5546f0770cba5e883dc34fa"
server_key_iv: "c53ac858221e8ab46b34e3e52f830007"
server_key_version: "encryptionkey@0"
I20260927 21:44:07.944165 18084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:44:07.947880 18099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:07.949374 18084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:44:07.949579 18084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "8c1766a4849c4cb386a5ffede17df2c5"
format_stamp: "Formatted at 2026-09-27 21:44:07 on dist-test-slave-25dq"
server_key: "b2fc85f9c5546f0770cba5e883dc34fa"
server_key_iv: "c53ac858221e8ab46b34e3e52f830007"
server_key_version: "encryptionkey@0"
I20260927 21:44:07.949807 18084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:07.977867 18084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:07.982767 18084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:07.983095 18084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:07.984166 18084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:07.985735 18084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:07.985842 18084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:07.986001 18084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:07.986083 18084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:08.022051 18084 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:35295
I20260927 21:44:08.022101 18211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:35295 every 8 connection(s)
I20260927 21:44:08.023415 18084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:44:08.026063 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 18084
I20260927 21:44:08.026330 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:44:08.027168 810 external_mini_cluster.cc:1428] Setting key 98d6afd3ef7e452d5ae18fc2a9f61ed0
Sep 27 21:44:08 dist-test-slave-25dq krb5kdc[17978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:08.047689 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:08.026071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:55769 (local address 127.0.202.190:44351)
0927 21:44:08.026721 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:08.026729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:08.027695 (+ 966us) server_negotiation.cc:407] Connection header received
0927 21:44:08.029919 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:08.029926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:08.030030 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:08.030236 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:08.032084 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.033484 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.035084 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.036001 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.036175 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:08.040404 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:08.040411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:08.040415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:08.040455 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:08.043709 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:08.044778 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:08.044792 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:08.044797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:08.044969 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:08.045516 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:08.045531 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:08.045534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:08.045855 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:08.046042 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:08.046907 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 21:44:08.047191 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":253,"threads_started":1}
I20260927 21:44:08.049016 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:08.026659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44351 (local address 127.0.202.129:55769)
0927 21:44:08.027491 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:08.027539 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:08.029611 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:08.030458 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:08.030466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:08.031185 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:08.031878 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:08.031896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.033647 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.033651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:08.034917 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:08.034928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.036246 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.036251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:08.036513 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:08.037229 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:08.037252 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:08.040076 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:08.043932 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:08.043944 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:08.043973 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:08.044609 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:08.045117 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:08.045123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:08.045128 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:08.045364 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:08.046027 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:08.046044 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:08.046455 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 21:44:08.046780 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 21:44:08.047427 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":363,"threads_started":1}
I20260927 21:44:08.050932 18212 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44351
I20260927 21:44:08.051276 18212 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:08.052093 18212 heartbeater.cc:511] Master 127.0.202.190:44351 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:44:08.054564 18023 ts_manager.cc:194] Registered new tserver with Master: 8c1766a4849c4cb386a5ffede17df2c5 (127.0.202.129:35295)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:44:08.056222 18023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:55769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:44:08.088444 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44351
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:08.431624 18218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:08.431914 18218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:08.432072 18218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:08.439332 18218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:08.439440 18218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:08.439625 18218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:44:08.449424 18218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:08.450793 18218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:08.452088 18218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:08.466328 18226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:08.466393 18223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:08.466347 18224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:08.467494 18218 server_base.cc:1034] running on GCE node
I20260927 21:44:08.468175 18218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:08.469205 18218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:08.470436 18218 hybrid_clock.cc:648] HybridClock initialized: now 1790545448470371 us; error 60 us; skew 500 ppm
I20260927 21:44:08.470779 18218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:08 dist-test-slave-25dq krb5kdc[17978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:08.474043 18218 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:44:08.475688 18218 webserver.cc:460] Webserver started at http://127.0.202.130:40673/ using document root <none> and password file <none>
I20260927 21:44:08.476526 18218 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:08.476697 18218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:08.477041 18218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:08.480767 18218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "b44a349852694e32a2e4624963200c7f"
format_stamp: "Formatted at 2026-09-27 21:44:08 on dist-test-slave-25dq"
server_key: "b0cadf52c4748dd33d17180a5b3df1c5"
server_key_iv: "2aed736c1185944591275a82ddb18b56"
server_key_version: "encryptionkey@0"
I20260927 21:44:08.481536 18218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "b44a349852694e32a2e4624963200c7f"
format_stamp: "Formatted at 2026-09-27 21:44:08 on dist-test-slave-25dq"
server_key: "b0cadf52c4748dd33d17180a5b3df1c5"
server_key_iv: "2aed736c1185944591275a82ddb18b56"
server_key_version: "encryptionkey@0"
I20260927 21:44:08.486953 18218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:44:08.490679 18233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:08.492172 18218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:44:08.492403 18218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "b44a349852694e32a2e4624963200c7f"
format_stamp: "Formatted at 2026-09-27 21:44:08 on dist-test-slave-25dq"
server_key: "b0cadf52c4748dd33d17180a5b3df1c5"
server_key_iv: "2aed736c1185944591275a82ddb18b56"
server_key_version: "encryptionkey@0"
I20260927 21:44:08.492588 18218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:08.522535 18218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:08.527388 18218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:08.527755 18218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:08.528868 18218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:08.530287 18218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:08.530359 18218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:08.530493 18218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:08.530548 18218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:08.565759 18218 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:37929
I20260927 21:44:08.565852 18345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:37929 every 8 connection(s)
I20260927 21:44:08.567080 18218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:44:08.574422 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 18218
I20260927 21:44:08.574607 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:44:08.575058 810 external_mini_cluster.cc:1428] Setting key 9ae0f578ee5ea7f9173d32207117dbef
Sep 27 21:44:08 dist-test-slave-25dq krb5kdc[17978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:08.588802 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:08.569762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:53503 (local address 127.0.202.190:44351)
0927 21:44:08.570347 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:08.570351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:08.571311 (+ 960us) server_negotiation.cc:407] Connection header received
0927 21:44:08.573281 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:08.573296 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:08.573390 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:08.573509 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:08.575164 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.576045 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.578054 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.578485 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.578586 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:08.582993 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:08.582998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:08.583001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:08.583034 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:08.585666 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:08.586454 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:08.586459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:08.586461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:08.586585 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:08.587027 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:08.587038 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:08.587041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:08.587297 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:08.587451 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:08.588150 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 21:44:08.588404 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":326,"threads_started":1}
I20260927 21:44:08.589804 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:08.570428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44351 (local address 127.0.202.130:53503)
0927 21:44:08.571145 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:08.571183 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:08.572993 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:08.573760 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:08.573768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:08.574296 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:08.574986 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:08.575004 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.576204 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.576214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:08.577897 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:08.577916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:08.578724 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:08.578733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:08.579113 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:08.579917 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:08.579954 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:08.582743 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:08.585828 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:08.585837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:08.585858 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:08.586309 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:08.586707 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:08.586712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:08.586716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:08.586904 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:08.587424 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:08.587432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:08.587795 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 21:44:08.588057 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 21:44:08.588651 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":279,"threads_started":1}
I20260927 21:44:08.591295 18346 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44351
I20260927 21:44:08.591652 18346 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:08.592564 18346 heartbeater.cc:511] Master 127.0.202.190:44351 requested a full tablet report, sending...
I20260927 21:44:08.594168 18023 ts_manager.cc:194] Registered new tserver with Master: b44a349852694e32a2e4624963200c7f (127.0.202.130:37929)
I20260927 21:44:08.594990 18023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:53503
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:44:08.637035 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44351
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:08.985877 18352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:08.986194 18352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:08.986363 18352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:08.994155 18352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:08.994277 18352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:08.994452 18352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:44:09.005103 18352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:09.006551 18352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:09.007936 18352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:09.022573 18358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:09.022707 18352 server_base.cc:1034] running on GCE node
W20260927 21:44:09.022856 18357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:09.023128 18360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:09.024098 18352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:09.025367 18352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:09.026639 18352 hybrid_clock.cc:648] HybridClock initialized: now 1790545449026566 us; error 58 us; skew 500 ppm
I20260927 21:44:09.027015 18352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:09 dist-test-slave-25dq krb5kdc[17978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:09.030743 18352 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:44:09.032574 18352 webserver.cc:460] Webserver started at http://127.0.202.131:33481/ using document root <none> and password file <none>
I20260927 21:44:09.033317 18352 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:09.033476 18352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:09.033797 18352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:09.036855 18352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "393af02c2cf8419ea063cf58e8e8b736"
format_stamp: "Formatted at 2026-09-27 21:44:09 on dist-test-slave-25dq"
server_key: "8ec89d0a94e43f732662dca7a3b6f332"
server_key_iv: "66fccb00c2ebfd58cb0b586bbd8b9cc8"
server_key_version: "encryptionkey@0"
I20260927 21:44:09.037667 18352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "393af02c2cf8419ea063cf58e8e8b736"
format_stamp: "Formatted at 2026-09-27 21:44:09 on dist-test-slave-25dq"
server_key: "8ec89d0a94e43f732662dca7a3b6f332"
server_key_iv: "66fccb00c2ebfd58cb0b586bbd8b9cc8"
server_key_version: "encryptionkey@0"
I20260927 21:44:09.043345 18352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:44:09.047479 18367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:09.049120 18352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:44:09.049321 18352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "393af02c2cf8419ea063cf58e8e8b736"
format_stamp: "Formatted at 2026-09-27 21:44:09 on dist-test-slave-25dq"
server_key: "8ec89d0a94e43f732662dca7a3b6f332"
server_key_iv: "66fccb00c2ebfd58cb0b586bbd8b9cc8"
server_key_version: "encryptionkey@0"
I20260927 21:44:09.049621 18352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:09.060503 18212 heartbeater.cc:503] Master 127.0.202.190:44351 was elected leader, sending a full tablet report...
I20260927 21:44:09.068883 18352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:09.073766 18352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:09.074072 18352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:09.075141 18352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:09.076639 18352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:09.076742 18352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:09.076893 18352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:09.076964 18352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:09.114526 18352 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:35345
I20260927 21:44:09.114566 18479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:35345 every 8 connection(s)
I20260927 21:44:09.116024 18352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:44:09.124670 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 18352
I20260927 21:44:09.124891 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:44:09.125397 810 external_mini_cluster.cc:1428] Setting key a4e2b720bece15590c48f68d899cd918
Sep 27 21:44:09 dist-test-slave-25dq krb5kdc[17978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:09.139076 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.119221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:52785 (local address 127.0.202.190:44351)
0927 21:44:09.119724 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.119729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.120705 (+ 976us) server_negotiation.cc:407] Connection header received
0927 21:44:09.122834 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.122838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.122932 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.123034 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:09.124828 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.125875 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.128245 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.128941 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.129044 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.133020 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:09.133025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:09.133029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:09.133070 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:09.135958 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.136787 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.136791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.136793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.136930 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.137390 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.137394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.137396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.137694 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:09.137822 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.138579 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.138766 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":275,"threads_started":1}
I20260927 21:44:09.140136 18483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.119691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44351 (local address 127.0.202.131:52785)
0927 21:44:09.120480 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.120543 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:09.122539 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:09.123272 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:09.123281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:09.123876 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:09.124624 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.124647 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.126081 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.126086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.128082 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.128095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.129188 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.129194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.129414 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.130219 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:09.130251 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:09.132798 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:09.136120 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:09.136126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:09.136142 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:09.136662 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:09.137036 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:09.137039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:09.137041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:09.137261 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:09.137814 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:09.137820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:09.138182 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 21:44:09.138417 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 21:44:09.138815 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260927 21:44:09.141710 18480 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44351
I20260927 21:44:09.142033 18480 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:09.142756 18480 heartbeater.cc:511] Master 127.0.202.190:44351 requested a full tablet report, sending...
I20260927 21:44:09.144277 18022 ts_manager.cc:194] Registered new tserver with Master: 393af02c2cf8419ea063cf58e8e8b736 (127.0.202.131:35345)
I20260927 21:44:09.145303 18022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:52785
I20260927 21:44:09.154711 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:44:09.173149 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.159813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.0.202.190:44351)
0927 21:44:09.160073 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.160084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.160492 (+ 408us) server_negotiation.cc:407] Connection header received
0927 21:44:09.160645 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.160649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.160719 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.160809 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:09.162234 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.163074 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.164455 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.164977 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.165120 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.167565 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:09.167570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:09.167574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:09.167609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:09.170424 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.171110 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.171113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.171116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.171242 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.171725 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.171729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.171731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.172064 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:09.172285 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.172705 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.172880 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:44:09.177762 18022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:44:09.185510 18022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:44:09.226943 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.199949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.0.202.129:35295)
0927 21:44:09.212338 (+ 12389us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.212354 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.214224 (+ 1870us) server_negotiation.cc:407] Connection header received
0927 21:44:09.214547 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.214556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.214835 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.215002 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:09.217539 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.219695 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.224226 (+ 4531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.225420 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.225563 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.225813 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.226398 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.226576 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12086,"thread_start_us":12404,"threads_started":1}
I20260927 21:44:09.227068 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.199368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.0.202.131:35345)
0927 21:44:09.214219 (+ 14851us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.214228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.214262 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:44:09.214367 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.214376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.214665 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.214937 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:09.215689 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.219897 (+ 4208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.223365 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.225013 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.225158 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.225404 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.226545 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.226721 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14655,"thread_start_us":14480,"threads_started":1}
I20260927 21:44:09.227741 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.199341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55448)
0927 21:44:09.214087 (+ 14746us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.214109 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:09.214342 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:09.216657 (+ 2315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:09.216664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:09.216704 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:09.217273 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.217289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.219977 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.219985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.224037 (+ 4052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.224050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.225966 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.225972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.226156 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.226196 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:44:09.227260 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0927 21:44:09.227381 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14599,"thread_start_us":13477,"threads_started":1}
I20260927 21:44:09.227783 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.199042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:35345 (local address 127.0.0.1:59514)
0927 21:44:09.213232 (+ 14190us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.213284 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:09.213508 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:09.214954 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:09.214962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:09.214997 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:09.215420 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.215433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.220119 (+ 4686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.220132 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.223173 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.223188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.225230 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.225236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.225697 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.225753 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:44:09.226281 (+ 528us) client_negotiation.cc:240] Negotiation successful
0927 21:44:09.227301 (+ 1020us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13921,"mutex_wait_us":851,"thread_start_us":15065,"threads_started":1}
I20260927 21:44:09.230368 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.215401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37929 (local address 127.0.0.1:50376)
0927 21:44:09.222011 (+ 6610us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.222032 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:09.222172 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:09.223558 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:09.223562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:09.223601 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:09.224039 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.224052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.225933 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.225942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.227789 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.227802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.229543 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.229549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.229720 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.229754 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:44:09.229846 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:44:09.229976 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6426,"thread_start_us":6857,"threads_started":1}
I20260927 21:44:09.230448 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.214819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.0.202.130:37929)
0927 21:44:09.222857 (+ 8038us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.222866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.222897 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:44:09.223015 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.223024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.223315 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.223551 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:09.224236 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.225804 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.228033 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.229314 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.229468 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.229652 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.229916 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.230104 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7872,"thread_start_us":7717,"threads_started":1}
I20260927 21:44:09.232097 18147 tablet_service.cc:1467] Processing CreateTablet for tablet 75a87375519e49bcb52e00d855b04a9c (DEFAULT_TABLE table=test-table [id=e8da170f8c3c4f2a8e017d0e04224cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:09.232095 18415 tablet_service.cc:1467] Processing CreateTablet for tablet 75a87375519e49bcb52e00d855b04a9c (DEFAULT_TABLE table=test-table [id=e8da170f8c3c4f2a8e017d0e04224cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:09.233657 18147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a87375519e49bcb52e00d855b04a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:09.234196 18415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a87375519e49bcb52e00d855b04a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:09.234651 18281 tablet_service.cc:1467] Processing CreateTablet for tablet 75a87375519e49bcb52e00d855b04a9c (DEFAULT_TABLE table=test-table [id=e8da170f8c3c4f2a8e017d0e04224cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:09.236128 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a87375519e49bcb52e00d855b04a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:09.244781 18499 tablet_bootstrap.cc:492] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: Bootstrap starting.
I20260927 21:44:09.248096 18499 tablet_bootstrap.cc:654] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:09.249485 18499 log.cc:824] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:09.251606 18502 tablet_bootstrap.cc:492] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Bootstrap starting.
I20260927 21:44:09.252085 18501 tablet_bootstrap.cc:492] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Bootstrap starting.
I20260927 21:44:09.252781 18499 tablet_bootstrap.cc:492] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: No bootstrap required, opened a new log
I20260927 21:44:09.253065 18499 ts_tablet_manager.cc:1397] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 21:44:09.255439 18502 tablet_bootstrap.cc:654] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:09.255720 18501 tablet_bootstrap.cc:654] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:09.257015 18501 log.cc:824] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:09.256868 18502 log.cc:824] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:09.264467 18502 tablet_bootstrap.cc:492] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: No bootstrap required, opened a new log
I20260927 21:44:09.264819 18502 ts_tablet_manager.cc:1397] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260927 21:44:09.264602 18499 raft_consensus.cc:348] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.265010 18499 raft_consensus.cc:374] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:09.265131 18501 tablet_bootstrap.cc:492] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: No bootstrap required, opened a new log
I20260927 21:44:09.265125 18499 raft_consensus.cc:729] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1766a4849c4cb386a5ffede17df2c5, State: Initialized, Role: FOLLOWER
I20260927 21:44:09.265425 18501 ts_tablet_manager.cc:1397] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 21:44:09.265777 18499 consensus_queue.cc:260] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.267611 18499 ts_tablet_manager.cc:1428] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
W20260927 21:44:09.275892 18213 tablet.cc:2369] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:09.277511 18502 raft_consensus.cc:348] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.277496 18501 raft_consensus.cc:348] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.278020 18502 raft_consensus.cc:374] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:09.278026 18501 raft_consensus.cc:374] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:09.278141 18501 raft_consensus.cc:729] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 393af02c2cf8419ea063cf58e8e8b736, State: Initialized, Role: FOLLOWER
I20260927 21:44:09.278154 18502 raft_consensus.cc:729] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44a349852694e32a2e4624963200c7f, State: Initialized, Role: FOLLOWER
I20260927 21:44:09.278869 18502 consensus_queue.cc:260] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.278869 18501 consensus_queue.cc:260] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.280098 18480 heartbeater.cc:503] Master 127.0.202.190:44351 was elected leader, sending a full tablet report...
I20260927 21:44:09.280557 18346 heartbeater.cc:503] Master 127.0.202.190:44351 was elected leader, sending a full tablet report...
I20260927 21:44:09.281239 18502 ts_tablet_manager.cc:1428] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260927 21:44:09.282056 18501 ts_tablet_manager.cc:1428] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260927 21:44:09.319155 18347 tablet.cc:2369] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:44:09.368106 18481 tablet.cc:2369] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:09.372551 18506 raft_consensus.cc:482] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:44:09.372880 18506 raft_consensus.cc:504] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.375063 18506 leader_election.cc:288] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c1766a4849c4cb386a5ffede17df2c5 (127.0.202.129:35295), b44a349852694e32a2e4624963200c7f (127.0.202.130:37929)
I20260927 21:44:09.382925 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.376085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:38785 (local address 127.0.202.130:37929)
0927 21:44:09.376270 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.376275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.376340 (+ 65us) server_negotiation.cc:407] Connection header received
0927 21:44:09.376420 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.376425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.376496 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.376621 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:09.377083 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.377989 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.380663 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.381896 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.382053 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.382122 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.382530 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.382665 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:44:09.382925 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.375599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:48183 (local address 127.0.202.129:35295)
0927 21:44:09.375817 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.375824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.376847 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 21:44:09.377038 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.377043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.377128 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.377225 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:09.377823 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.378617 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.380587 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.381850 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.382034 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.382122 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.382530 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.382665 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:44:09.383085 18483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.375353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37929 (local address 127.0.202.131:38785)
0927 21:44:09.375849 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.375874 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:09.376024 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:09.376643 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:09.376648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:09.376680 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:09.376892 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.376904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.378134 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.378140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.380493 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.380509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.382068 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.382075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.382325 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.382384 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 21:44:09.382498 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:44:09.382694 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":105}
I20260927 21:44:09.383170 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.375399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.202.131:48183)
0927 21:44:09.376739 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.376753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:09.376894 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:09.377257 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:09.377262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:09.377288 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:09.377651 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.377663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.378771 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.378776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.380435 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:09.380449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.382081 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.382086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:09.382325 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.382382 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:44:09.382491 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:44:09.382694 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":73,"thread_start_us":951,"threads_started":1}
I20260927 21:44:09.384191 18301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a87375519e49bcb52e00d855b04a9c" candidate_uuid: "393af02c2cf8419ea063cf58e8e8b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44a349852694e32a2e4624963200c7f" is_pre_election: true
I20260927 21:44:09.384191 18167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a87375519e49bcb52e00d855b04a9c" candidate_uuid: "393af02c2cf8419ea063cf58e8e8b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1766a4849c4cb386a5ffede17df2c5" is_pre_election: true
I20260927 21:44:09.384769 18301 raft_consensus.cc:2454] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 393af02c2cf8419ea063cf58e8e8b736 in term 0.
I20260927 21:44:09.384783 18167 raft_consensus.cc:2454] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 393af02c2cf8419ea063cf58e8e8b736 in term 0.
I20260927 21:44:09.385489 18368 leader_election.cc:302] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 393af02c2cf8419ea063cf58e8e8b736, 8c1766a4849c4cb386a5ffede17df2c5; no voters:
I20260927 21:44:09.385922 18506 raft_consensus.cc:2781] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:44:09.386123 18506 raft_consensus.cc:482] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:44:09.386251 18506 raft_consensus.cc:3037] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:09.388871 18506 raft_consensus.cc:504] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.389566 18506 leader_election.cc:288] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [CANDIDATE]: Term 1 election: Requested vote from peers 8c1766a4849c4cb386a5ffede17df2c5 (127.0.202.129:35295), b44a349852694e32a2e4624963200c7f (127.0.202.130:37929)
I20260927 21:44:09.390347 18167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a87375519e49bcb52e00d855b04a9c" candidate_uuid: "393af02c2cf8419ea063cf58e8e8b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1766a4849c4cb386a5ffede17df2c5"
I20260927 21:44:09.390347 18301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a87375519e49bcb52e00d855b04a9c" candidate_uuid: "393af02c2cf8419ea063cf58e8e8b736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44a349852694e32a2e4624963200c7f"
I20260927 21:44:09.390610 18167 raft_consensus.cc:3037] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:09.390678 18301 raft_consensus.cc:3037] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:09.394326 18167 raft_consensus.cc:2454] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 393af02c2cf8419ea063cf58e8e8b736 in term 1.
I20260927 21:44:09.394326 18301 raft_consensus.cc:2454] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 393af02c2cf8419ea063cf58e8e8b736 in term 1.
I20260927 21:44:09.394888 18368 leader_election.cc:302] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 393af02c2cf8419ea063cf58e8e8b736, 8c1766a4849c4cb386a5ffede17df2c5, b44a349852694e32a2e4624963200c7f; no voters:
I20260927 21:44:09.395315 18506 raft_consensus.cc:2781] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:09.395787 18506 raft_consensus.cc:686] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 1 LEADER]: Becoming Leader. State: Replica: 393af02c2cf8419ea063cf58e8e8b736, State: Running, Role: LEADER
I20260927 21:44:09.396516 18506 consensus_queue.cc:237] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } }
I20260927 21:44:09.401324 18023 catalog_manager.cc:5495] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 reported cstate change: term changed from 0 to 1, leader changed from <none> to 393af02c2cf8419ea063cf58e8e8b736 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "393af02c2cf8419ea063cf58e8e8b736" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "393af02c2cf8419ea063cf58e8e8b736" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 35345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 } health_report { overall_health: UNKNOWN } } }
I20260927 21:44:09.423008 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.417168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.0.202.131:35345)
0927 21:44:09.417426 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.417434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.417457 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:44:09.417728 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.417732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.417814 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.417941 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:44:09.418292 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.419049 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.420696 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.421084 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.421177 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.421769 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:44:09.422275 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:44:09.422397 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.422618 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.422683 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 21:44:09.431757 18167 raft_consensus.cc:1261] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 1 FOLLOWER]: Refusing update from remote peer 393af02c2cf8419ea063cf58e8e8b736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:09.431821 18301 raft_consensus.cc:1261] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 1 FOLLOWER]: Refusing update from remote peer 393af02c2cf8419ea063cf58e8e8b736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:09.433050 18509 consensus_queue.cc:1035] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1766a4849c4cb386a5ffede17df2c5" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:09.433416 18506 consensus_queue.cc:1035] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44a349852694e32a2e4624963200c7f" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:09.445693 18512 mvcc.cc:205] Tried to move back new op lower bound from 7334074160861323264 to 7334074160732135424. Current Snapshot: MvccSnapshot[applied={T|T < 7334074160861323264}]
I20260927 21:44:09.460922 18023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:09.461246 18023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:09.465068 18023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5db2b46f54d24cfb8e6c80d9f55e8469: Sending DeleteTablet for 3 replicas of tablet 75a87375519e49bcb52e00d855b04a9c
I20260927 21:44:09.466390 18415 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a87375519e49bcb52e00d855b04a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:09 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:59514
I20260927 21:44:09.466387 18281 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a87375519e49bcb52e00d855b04a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:09 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:50376
I20260927 21:44:09.466435 18147 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a87375519e49bcb52e00d855b04a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:09 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:55448
I20260927 21:44:09.468750 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 18084
I20260927 21:44:09.505049 18524 tablet_replica.cc:330] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: stopping tablet replica
I20260927 21:44:09.505772 18522 tablet_replica.cc:330] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: stopping tablet replica
I20260927 21:44:09.506296 18522 raft_consensus.cc:2229] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:44:09.507050 18522 raft_consensus.cc:2258] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:09.510358 18522 ts_tablet_manager.cc:1905] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:09.512161 18524 raft_consensus.cc:2229] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:09.513139 18524 raft_consensus.cc:2258] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 21:44:09 dist-test-slave-25dq krb5kdc[17978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:44:09.518653 18524 ts_tablet_manager.cc:1905] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:09.519495 18523 tablet_replica.cc:330] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: stopping tablet replica
I20260927 21:44:09.520761 18523 raft_consensus.cc:2229] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:09.521420 18523 raft_consensus.cc:2258] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:09.524159 18522 ts_tablet_manager.cc:1918] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:09.524376 18522 log.cc:1196] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/75a87375519e49bcb52e00d855b04a9c
I20260927 21:44:09.524959 18522 ts_tablet_manager.cc:1939] T 75a87375519e49bcb52e00d855b04a9c P 393af02c2cf8419ea063cf58e8e8b736: Deleting consensus metadata
I20260927 21:44:09.526264 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.470237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.0.202.129:35295)
0927 21:44:09.470427 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.470433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.497854 (+ 27421us) server_negotiation.cc:407] Connection header received
0927 21:44:09.498398 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.498405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.498503 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.498584 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:09.500517 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.501752 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.507618 (+ 5866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.508405 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.508542 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.517305 (+ 8763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:09.517329 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:09.517347 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:09.517411 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:09.522136 (+ 4725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.523199 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.523227 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.523230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.523416 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.523900 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.523905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.523908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.524444 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:09.524750 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.525602 (+ 852us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.525872 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:44:09.526876 18010 catalog_manager.cc:4841] TS 393af02c2cf8419ea063cf58e8e8b736 (127.0.202.131:35345): tablet 75a87375519e49bcb52e00d855b04a9c (table test-table [id=e8da170f8c3c4f2a8e017d0e04224cc1]) successfully deleted
I20260927 21:44:09.527872 18207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:09.529131 18524 ts_tablet_manager.cc:1918] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:09.529405 18524 log.cc:1196] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/75a87375519e49bcb52e00d855b04a9c
I20260927 21:44:09.530171 18524 ts_tablet_manager.cc:1939] T 75a87375519e49bcb52e00d855b04a9c P b44a349852694e32a2e4624963200c7f: Deleting consensus metadata
I20260927 21:44:09.532727 18010 catalog_manager.cc:4841] TS b44a349852694e32a2e4624963200c7f (127.0.202.130:37929): tablet 75a87375519e49bcb52e00d855b04a9c (table test-table [id=e8da170f8c3c4f2a8e017d0e04224cc1]) successfully deleted
I20260927 21:44:09.528623 18523 ts_tablet_manager.cc:1905] T 75a87375519e49bcb52e00d855b04a9c P 8c1766a4849c4cb386a5ffede17df2c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:09.636049 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 18084
W20260927 21:44:09.659875 18007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.202.129:35295 (error 108)
I20260927 21:44:09.660130 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 18218
I20260927 21:44:09.662225 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.661731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55474)
0927 21:44:09.661914 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.662030 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 21:44:09.662654 18007 catalog_manager.cc:4584] TS 8c1766a4849c4cb386a5ffede17df2c5 (127.0.202.129:35295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75a87375519e49bcb52e00d855b04a9c: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Sep 27 21:44:09 dist-test-slave-25dq krb5kdc[17978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:44:09.679972 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.661332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.0.202.130:37929)
0927 21:44:09.661528 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.661535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.661558 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:44:09.661627 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.661633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.661729 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.661847 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:09.663632 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.664921 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.666461 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.667139 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.667284 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.672731 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:09.672754 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:09.672774 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:09.672826 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:09.676862 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.677689 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.677699 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.677703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.677877 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.678265 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.678270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.678273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.678764 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:09.679078 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.679328 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.679568 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:44:09.680729 18341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:09.716455 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.715826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55478)
0927 21:44:09.716075 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.716201 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 21:44:09.717638 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.717248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55486)
0927 21:44:09.717394 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.717484 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:44:09.772943 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 18218
I20260927 21:44:09.778486 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.778007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55500)
0927 21:44:09.778246 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.778333 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 21:44:09.779613 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.779267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55504)
0927 21:44:09.779420 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.779497 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:44:09.797466 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 18352
Sep 27 21:44:09 dist-test-slave-25dq krb5kdc[17978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:44:09.816668 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.798322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.0.202.131:35345)
0927 21:44:09.798518 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:09.798524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:09.798572 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:44:09.798764 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:09.798769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:09.798856 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:09.799011 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:09.800519 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.801708 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.803192 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:09.803871 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:09.804043 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:09.809223 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:09.809244 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:09.809266 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:09.809322 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:09.813465 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.814330 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.814339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.814343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.814536 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:09.814912 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:09.814926 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:09.814930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:09.815433 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:09.815779 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:09.816004 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 21:44:09.816237 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:44:09.817464 18475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:09.860880 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.860344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55512)
0927 21:44:09.860567 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.860668 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:44:09.861986 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:09.861624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55522)
0927 21:44:09.861814 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:09.861882 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 21:44:09.946643 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 18352
I20260927 21:44:09.966719 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 17991
I20260927 21:44:09.967880 18053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:10.080093 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:10.079442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35295 (local address 127.0.0.1:55528)
0927 21:44:10.079734 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:10.079851 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 21:44:10.080269 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 17991
2026-09-27T21:44:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4594 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T21:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:10Z Disabled control of system clock
I20260927 21:44:10.159868 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40841
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:44:10.503497 18534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:10.503818 18534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:10.503965 18534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:10.511575 18534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:10.511706 18534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:10.511780 18534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:10.511935 18534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:10.512017 18534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:10.521543 18534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:10.522934 18534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:10.524938 18534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:10.537966 18539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:10.538031 18540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:10.538074 18542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:10.538720 18534 server_base.cc:1034] running on GCE node
I20260927 21:44:10.539587 18534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:10.541144 18534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:10.542462 18534 hybrid_clock.cc:648] HybridClock initialized: now 1790545450542362 us; error 48 us; skew 500 ppm
I20260927 21:44:10.542903 18534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:10.545585 18534 webserver.cc:460] Webserver started at http://127.0.202.190:33231/ using document root <none> and password file <none>
I20260927 21:44:10.546401 18534 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:10.546548 18534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:10.546926 18534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:10.550261 18534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "f83f8012e5bd4d91a194e894dbb26c55"
format_stamp: "Formatted at 2026-09-27 21:44:10 on dist-test-slave-25dq"
server_key: "266f2e764503549f69017d0b6f0908ad"
server_key_iv: "ee431d461d46b9d9d0dc76cdfa289fe2"
server_key_version: "encryptionkey@0"
I20260927 21:44:10.551159 18534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "f83f8012e5bd4d91a194e894dbb26c55"
format_stamp: "Formatted at 2026-09-27 21:44:10 on dist-test-slave-25dq"
server_key: "266f2e764503549f69017d0b6f0908ad"
server_key_iv: "ee431d461d46b9d9d0dc76cdfa289fe2"
server_key_version: "encryptionkey@0"
I20260927 21:44:10.556952 18534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:44:10.560999 18548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:10.562670 18534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:44:10.562887 18534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "f83f8012e5bd4d91a194e894dbb26c55"
format_stamp: "Formatted at 2026-09-27 21:44:10 on dist-test-slave-25dq"
server_key: "266f2e764503549f69017d0b6f0908ad"
server_key_iv: "ee431d461d46b9d9d0dc76cdfa289fe2"
server_key_version: "encryptionkey@0"
I20260927 21:44:10.563119 18534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:10.577687 18534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:10.578606 18534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:10.578914 18534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:10.598726 18534 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40841
I20260927 21:44:10.598732 18599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40841 every 8 connection(s)
I20260927 21:44:10.600153 18534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:10.604429 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:10.606357 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 18534
I20260927 21:44:10.606549 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:10.607142 810 external_mini_cluster.cc:1428] Setting key 0c45045c6f297eb5432b572145232287
I20260927 21:44:10.619073 18600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55: Bootstrap starting.
I20260927 21:44:10.621312 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:10.609386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.0.202.190:40841)
0927 21:44:10.610443 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:10.610455 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:10.610499 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:44:10.612124 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:10.612147 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:10.612729 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:10.613143 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:10.613733 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:10.615004 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:10.616429 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:10.617135 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:10.617330 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:10.618823 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:10.618832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:10.618846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:10.619470 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:10.619552 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:10.619578 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:10.619717 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:10.619930 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:44:10.620278 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":392,"threads_started":1}
I20260927 21:44:10.622856 18600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:10.624208 18600 log.cc:824] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:10.627468 18600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55: No bootstrap required, opened a new log
I20260927 21:44:10.640692 18600 raft_consensus.cc:348] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83f8012e5bd4d91a194e894dbb26c55" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40841 } }
I20260927 21:44:10.641129 18600 raft_consensus.cc:374] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:10.641208 18600 raft_consensus.cc:729] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f8012e5bd4d91a194e894dbb26c55, State: Initialized, Role: FOLLOWER
I20260927 21:44:10.641884 18600 consensus_queue.cc:260] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83f8012e5bd4d91a194e894dbb26c55" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40841 } }
I20260927 21:44:10.642207 18600 raft_consensus.cc:388] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:10.642397 18600 raft_consensus.cc:482] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:10.642608 18600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:10.645156 18600 raft_consensus.cc:504] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83f8012e5bd4d91a194e894dbb26c55" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40841 } }
I20260927 21:44:10.645750 18600 leader_election.cc:302] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f83f8012e5bd4d91a194e894dbb26c55; no voters:
I20260927 21:44:10.646579 18600 leader_election.cc:288] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:10.646764 18605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:10.647764 18605 raft_consensus.cc:686] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [term 1 LEADER]: Becoming Leader. State: Replica: f83f8012e5bd4d91a194e894dbb26c55, State: Running, Role: LEADER
I20260927 21:44:10.648504 18605 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83f8012e5bd4d91a194e894dbb26c55" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40841 } }
I20260927 21:44:10.649127 18600 sys_catalog.cc:564] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:10.651228 18607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83f8012e5bd4d91a194e894dbb26c55. Latest consensus state: current_term: 1 leader_uuid: "f83f8012e5bd4d91a194e894dbb26c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83f8012e5bd4d91a194e894dbb26c55" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40841 } } }
I20260927 21:44:10.651772 18607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:10.651795 18606 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f83f8012e5bd4d91a194e894dbb26c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83f8012e5bd4d91a194e894dbb26c55" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40841 } } }
I20260927 21:44:10.652195 18606 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:10.654733 18610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:10.660169 18610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:10.670145 18610 catalog_manager.cc:1347] Generated new cluster ID: 1f3ac864e5be4a5d9d18f32afcb53436
I20260927 21:44:10.670311 18610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:10.697012 18610 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:10.698443 18610 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:10.711243 18610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55: Generated new TSK 0
I20260927 21:44:10.712453 18610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:10.730865 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40841
--builtin_ntp_servers=127.0.202.148:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:44:11.095459 18624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:11.095820 18624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:11.095968 18624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:11.103360 18624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:11.103479 18624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:11.103590 18624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:44:11.113251 18624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:11.114578 18624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:11.116580 18624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:11.130568 18632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:11.130606 18630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:11.130501 18629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:11.131319 18624 server_base.cc:1034] running on GCE node
I20260927 21:44:11.131965 18624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:11.133085 18624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:11.134295 18624 hybrid_clock.cc:648] HybridClock initialized: now 1790545451134252 us; error 37 us; skew 500 ppm
I20260927 21:44:11.134683 18624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:11.137173 18624 webserver.cc:460] Webserver started at http://127.0.202.129:44011/ using document root <none> and password file <none>
I20260927 21:44:11.137909 18624 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:11.138022 18624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:11.138376 18624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:11.141383 18624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "8dba2b72159c401ba07e9a576133b87f"
format_stamp: "Formatted at 2026-09-27 21:44:11 on dist-test-slave-25dq"
server_key: "cb3a1b15c9dfc2646bce4fcc21ac7455"
server_key_iv: "099970c400a3910dae8e9cc8e73e88dc"
server_key_version: "encryptionkey@0"
I20260927 21:44:11.142185 18624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "8dba2b72159c401ba07e9a576133b87f"
format_stamp: "Formatted at 2026-09-27 21:44:11 on dist-test-slave-25dq"
server_key: "cb3a1b15c9dfc2646bce4fcc21ac7455"
server_key_iv: "099970c400a3910dae8e9cc8e73e88dc"
server_key_version: "encryptionkey@0"
I20260927 21:44:11.147748 18624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:44:11.151734 18638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:11.153332 18624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:44:11.153522 18624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "8dba2b72159c401ba07e9a576133b87f"
format_stamp: "Formatted at 2026-09-27 21:44:11 on dist-test-slave-25dq"
server_key: "cb3a1b15c9dfc2646bce4fcc21ac7455"
server_key_iv: "099970c400a3910dae8e9cc8e73e88dc"
server_key_version: "encryptionkey@0"
I20260927 21:44:11.153757 18624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:11.179100 18624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:11.180034 18624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:11.180425 18624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:11.181501 18624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:11.183053 18624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:11.183179 18624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:11.183319 18624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:11.183413 18624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:11.219893 18624 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:43035
I20260927 21:44:11.219991 18750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:43035 every 8 connection(s)
I20260927 21:44:11.221318 18624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:44:11.228888 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 18624
I20260927 21:44:11.229106 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:44:11.229643 810 external_mini_cluster.cc:1428] Setting key e110313fe3f5e84e41e465e60b865e7f
I20260927 21:44:11.232579 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40841
--builtin_ntp_servers=127.0.202.148:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:44:11.235777 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:11.224369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:41895 (local address 127.0.202.190:40841)
0927 21:44:11.224915 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:11.224922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:11.226436 (+ 1514us) server_negotiation.cc:407] Connection header received
0927 21:44:11.228226 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:11.228230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:11.228385 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:11.228520 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:11.230193 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.230964 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.232681 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.233175 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.233321 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:11.234899 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:11.234904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:11.234907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:11.234925 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:11.234968 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:11.234984 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:11.235087 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:11.235319 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 21:44:11.235425 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":278,"threads_started":1}
I20260927 21:44:11.236918 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:11.225138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40841 (local address 127.0.202.129:41895)
0927 21:44:11.226145 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:11.226183 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:11.227919 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:11.228904 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:11.228919 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:11.229066 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:11.229967 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:11.230003 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.231156 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.231164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:11.232528 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:11.232542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.233423 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.233430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:11.233680 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:11.234531 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:11.234534 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:11.234577 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:11.234668 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:11.235088 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:11.235102 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:11.235112 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:44:11.235310 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 21:44:11.235753 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":285,"threads_started":1}
I20260927 21:44:11.238603 18751 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40841
I20260927 21:44:11.238940 18751 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:11.239753 18751 heartbeater.cc:511] Master 127.0.202.190:40841 requested a full tablet report, sending...
I20260927 21:44:11.242254 18565 ts_manager.cc:194] Registered new tserver with Master: 8dba2b72159c401ba07e9a576133b87f (127.0.202.129:43035)
I20260927 21:44:11.243794 18565 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.129:41895
W20260927 21:44:11.576655 18755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:11.577013 18755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:11.577168 18755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:11.584929 18755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:11.585081 18755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:11.585259 18755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:44:11.594828 18755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:11.596232 18755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:11.598263 18755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:11.612991 18760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:11.613510 18761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:11.615294 18763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:11.616032 18755 server_base.cc:1034] running on GCE node
I20260927 21:44:11.616823 18755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:11.618031 18755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:11.619277 18755 hybrid_clock.cc:648] HybridClock initialized: now 1790545451619220 us; error 46 us; skew 500 ppm
I20260927 21:44:11.619660 18755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:11.622699 18755 webserver.cc:460] Webserver started at http://127.0.202.130:34071/ using document root <none> and password file <none>
I20260927 21:44:11.623456 18755 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:11.623558 18755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:11.623860 18755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:11.627609 18755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "78bf055d9c69450cb44e867599f13947"
format_stamp: "Formatted at 2026-09-27 21:44:11 on dist-test-slave-25dq"
server_key: "5ce7fcca712de8ec9c0f6f3c2e6da775"
server_key_iv: "be39999c76a55ff0db0e0bc7c9bc89b8"
server_key_version: "encryptionkey@0"
I20260927 21:44:11.628762 18755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "78bf055d9c69450cb44e867599f13947"
format_stamp: "Formatted at 2026-09-27 21:44:11 on dist-test-slave-25dq"
server_key: "5ce7fcca712de8ec9c0f6f3c2e6da775"
server_key_iv: "be39999c76a55ff0db0e0bc7c9bc89b8"
server_key_version: "encryptionkey@0"
I20260927 21:44:11.635782 18755 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260927 21:44:11.639771 18769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:11.641530 18755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:44:11.641767 18755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "78bf055d9c69450cb44e867599f13947"
format_stamp: "Formatted at 2026-09-27 21:44:11 on dist-test-slave-25dq"
server_key: "5ce7fcca712de8ec9c0f6f3c2e6da775"
server_key_iv: "be39999c76a55ff0db0e0bc7c9bc89b8"
server_key_version: "encryptionkey@0"
I20260927 21:44:11.642024 18755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:11.653879 18755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:11.654740 18755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:11.655054 18755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:11.656144 18755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:11.657917 18755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:11.658041 18755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:11.658177 18755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:11.658262 18755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:11.695554 18755 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:37509
I20260927 21:44:11.695685 18881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:37509 every 8 connection(s)
I20260927 21:44:11.696964 18755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:44:11.705017 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 18755
I20260927 21:44:11.705209 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:44:11.705726 810 external_mini_cluster.cc:1428] Setting key 76cdd6e05b07c2c6b625451604478d5f
I20260927 21:44:11.709125 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40841
--builtin_ntp_servers=127.0.202.148:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:44:11.711537 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:11.699758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:48117 (local address 127.0.202.190:40841)
0927 21:44:11.699934 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:11.699938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:11.701269 (+ 1331us) server_negotiation.cc:407] Connection header received
0927 21:44:11.703105 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:11.703111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:11.703204 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:11.703317 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:11.704479 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.705620 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.707933 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.708539 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.708683 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:11.710342 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:11.710352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:11.710360 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:11.710387 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:11.710433 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:11.710452 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:11.710561 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:11.710961 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 21:44:11.711088 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:44:11.712790 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:11.700153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40841 (local address 127.0.202.130:48117)
0927 21:44:11.701106 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:11.701145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:11.702808 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:11.703560 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:11.703568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:11.703669 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:11.704270 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:11.704286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.705820 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.705827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:11.707768 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:11.707788 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:11.708787 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:11.708793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:11.709022 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:11.709961 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:11.709963 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:11.710006 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:11.710118 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:11.710702 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:11.710711 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:11.710718 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:44:11.710912 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 21:44:11.711426 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":287,"threads_started":1}
I20260927 21:44:11.714752 18882 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40841
I20260927 21:44:11.715130 18882 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:11.715976 18882 heartbeater.cc:511] Master 127.0.202.190:40841 requested a full tablet report, sending...
I20260927 21:44:11.717639 18565 ts_manager.cc:194] Registered new tserver with Master: 78bf055d9c69450cb44e867599f13947 (127.0.202.130:37509)
I20260927 21:44:11.718493 18565 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.130:48117
W20260927 21:44:12.059161 18885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:12.059552 18885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:12.059725 18885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:12.068113 18885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:12.068277 18885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:12.068509 18885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:44:12.078104 18885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:40841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:12.079849 18885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:12.081929 18885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:12.096407 18890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:12.096335 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:12.096436 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:12.097795 18885 server_base.cc:1034] running on GCE node
I20260927 21:44:12.098543 18885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:12.099716 18885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:12.100966 18885 hybrid_clock.cc:648] HybridClock initialized: now 1790545452100920 us; error 56 us; skew 500 ppm
I20260927 21:44:12.101305 18885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:12.103645 18885 webserver.cc:460] Webserver started at http://127.0.202.131:36025/ using document root <none> and password file <none>
I20260927 21:44:12.104404 18885 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:12.104527 18885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:12.104844 18885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:12.107751 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "e41dc5df26014106ba7b7968a834dac9"
format_stamp: "Formatted at 2026-09-27 21:44:12 on dist-test-slave-25dq"
server_key: "690278bdea21753488085a7d51a18944"
server_key_iv: "a024132253bb624b0f91f8efa8ae07a7"
server_key_version: "encryptionkey@0"
I20260927 21:44:12.108594 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "e41dc5df26014106ba7b7968a834dac9"
format_stamp: "Formatted at 2026-09-27 21:44:12 on dist-test-slave-25dq"
server_key: "690278bdea21753488085a7d51a18944"
server_key_iv: "a024132253bb624b0f91f8efa8ae07a7"
server_key_version: "encryptionkey@0"
I20260927 21:44:12.113749 18885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:44:12.117416 18899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:12.118824 18885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:44:12.119000 18885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "e41dc5df26014106ba7b7968a834dac9"
format_stamp: "Formatted at 2026-09-27 21:44:12 on dist-test-slave-25dq"
server_key: "690278bdea21753488085a7d51a18944"
server_key_iv: "a024132253bb624b0f91f8efa8ae07a7"
server_key_version: "encryptionkey@0"
I20260927 21:44:12.119264 18885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:12.141702 18885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:12.142586 18885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:12.142892 18885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:12.143921 18885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:12.145583 18885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:12.145716 18885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:12.145812 18885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:12.145869 18885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:12.181252 18885 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:38805
I20260927 21:44:12.181314 19011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:38805 every 8 connection(s)
I20260927 21:44:12.182606 18885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:44:12.185715 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 18885
I20260927 21:44:12.185889 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:44:12.186372 810 external_mini_cluster.cc:1428] Setting key 43285297c00b5f1ea22270577b8ba36e
I20260927 21:44:12.196493 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.185620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:45063 (local address 127.0.202.190:40841)
0927 21:44:12.185880 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.185889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.187716 (+ 1827us) server_negotiation.cc:407] Connection header received
0927 21:44:12.189712 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.189716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.189785 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.189897 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:12.191122 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.191891 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.193649 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.194066 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.194180 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.195599 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:12.195603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:12.195607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:12.195627 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:12.195659 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:12.195671 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:12.195750 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.195996 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.196099 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:44:12.197515 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.186291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:40841 (local address 127.0.202.131:45063)
0927 21:44:12.187542 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:12.187592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:12.189451 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:12.190190 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:12.190200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:12.190303 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:12.190954 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.190973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.192055 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.192059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.193447 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.193461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.194280 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.194285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.194503 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.195280 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:12.195282 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:12.195322 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:12.195406 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:12.195764 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:12.195772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:12.195778 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:44:12.195928 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 21:44:12.196232 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":388,"threads_started":1}
I20260927 21:44:12.199072 19012 heartbeater.cc:347] Connected to a master server at 127.0.202.190:40841
I20260927 21:44:12.199383 19012 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:12.200187 19012 heartbeater.cc:511] Master 127.0.202.190:40841 requested a full tablet report, sending...
I20260927 21:44:12.201830 18564 ts_manager.cc:194] Registered new tserver with Master: e41dc5df26014106ba7b7968a834dac9 (127.0.202.131:38805)
I20260927 21:44:12.202715 18564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.131:45063
I20260927 21:44:12.214823 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:44:12.225690 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.219224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.0.202.190:40841)
0927 21:44:12.219493 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.219502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.219913 (+ 411us) server_negotiation.cc:407] Connection header received
0927 21:44:12.220123 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.220127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.220192 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.220296 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:12.220820 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.221548 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.223051 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.223590 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.223764 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.224961 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:12.224972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:12.224975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:12.224995 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:12.225040 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:12.225053 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:12.225148 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.225287 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.225385 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 21:44:12.229869 18564 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:44:12.237639 18564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:44:12.246932 18751 heartbeater.cc:503] Master 127.0.202.190:40841 was elected leader, sending a full tablet report...
I20260927 21:44:12.264441 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.250343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.0.202.129:43035)
0927 21:44:12.250975 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.250986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.251441 (+ 455us) server_negotiation.cc:407] Connection header received
0927 21:44:12.251800 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.251812 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.252146 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.254101 (+ 1955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:12.254412 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.256989 (+ 2577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.261149 (+ 4160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.263033 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.263206 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.263552 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.263806 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.264004 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":320,"threads_started":1}
I20260927 21:44:12.264422 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.250107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:43035 (local address 127.0.0.1:50234)
0927 21:44:12.251302 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:12.251338 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:12.251535 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:12.252432 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:12.252444 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:12.252490 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:12.252922 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.252937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.257214 (+ 4277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.257224 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.260812 (+ 3588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.260845 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.263253 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.263261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.263528 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.263609 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 21:44:12.263756 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 21:44:12.263988 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":861,"threads_started":1}
I20260927 21:44:12.269606 18686 tablet_service.cc:1467] Processing CreateTablet for tablet df7d8974e00847bf9bc7a1e8fdfe60f1 (DEFAULT_TABLE table=test-table [id=e2246108b5bf4ccca1a9ebc3501e86ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:12.270334 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.251649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37509 (local address 127.0.0.1:48948)
0927 21:44:12.252207 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:12.252229 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:12.252389 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:12.262647 (+ 10258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:12.262652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:12.262682 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:12.263144 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.263158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.265502 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.265514 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.267484 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.267500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.269503 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.269510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.269690 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.269726 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:44:12.269826 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:44:12.269978 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":251,"threads_started":1}
I20260927 21:44:12.270493 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.251969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.0.202.130:37509)
0927 21:44:12.262012 (+ 10043us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.262022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.262050 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:44:12.262144 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.262151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.262408 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.263391 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:12.263672 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.265169 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.267738 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.269280 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.269436 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.269698 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.269898 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.270100 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9864,"thread_start_us":9494,"threads_started":1}
I20260927 21:44:12.271694 18686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7d8974e00847bf9bc7a1e8fdfe60f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:12.271867 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.252271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:38805 (local address 127.0.0.1:46528)
0927 21:44:12.260955 (+ 8684us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:12.260983 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:12.261166 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:12.262335 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:12.262353 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:12.262386 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:12.262844 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.262858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.265129 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.265134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.268486 (+ 3352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.268501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.271015 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.271020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.271141 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.271166 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:44:12.271285 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:44:12.271486 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8479,"mutex_wait_us":109,"thread_start_us":7450,"threads_started":1}
I20260927 21:44:12.272014 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.252789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.0.202.131:38805)
0927 21:44:12.253653 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.253674 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.261145 (+ 7471us) server_negotiation.cc:407] Connection header received
0927 21:44:12.261382 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.261391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.261741 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.262028 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:12.263071 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.264953 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.268717 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.270422 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.270594 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.270905 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.271374 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.271587 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":447,"threads_started":1}
I20260927 21:44:12.282294 18817 tablet_service.cc:1467] Processing CreateTablet for tablet df7d8974e00847bf9bc7a1e8fdfe60f1 (DEFAULT_TABLE table=test-table [id=e2246108b5bf4ccca1a9ebc3501e86ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:12.283875 18817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7d8974e00847bf9bc7a1e8fdfe60f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:12.288391 19030 tablet_bootstrap.cc:492] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Bootstrap starting.
I20260927 21:44:12.289506 18947 tablet_service.cc:1467] Processing CreateTablet for tablet df7d8974e00847bf9bc7a1e8fdfe60f1 (DEFAULT_TABLE table=test-table [id=e2246108b5bf4ccca1a9ebc3501e86ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:12.291198 18947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7d8974e00847bf9bc7a1e8fdfe60f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:12.292914 19030 tablet_bootstrap.cc:654] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:12.294119 19030 log.cc:824] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:12.297462 19030 tablet_bootstrap.cc:492] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: No bootstrap required, opened a new log
I20260927 21:44:12.297844 19030 ts_tablet_manager.cc:1397] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260927 21:44:12.302347 19032 tablet_bootstrap.cc:492] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Bootstrap starting.
I20260927 21:44:12.305553 19033 tablet_bootstrap.cc:492] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Bootstrap starting.
I20260927 21:44:12.306528 19032 tablet_bootstrap.cc:654] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:12.308012 19032 log.cc:824] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:12.311175 19033 tablet_bootstrap.cc:654] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:12.311038 19030 raft_consensus.cc:348] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.311692 19030 raft_consensus.cc:374] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:12.311836 19030 raft_consensus.cc:729] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dba2b72159c401ba07e9a576133b87f, State: Initialized, Role: FOLLOWER
I20260927 21:44:12.312845 19033 log.cc:824] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:12.312883 19030 consensus_queue.cc:260] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.317175 19033 tablet_bootstrap.cc:492] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: No bootstrap required, opened a new log
I20260927 21:44:12.317226 19032 tablet_bootstrap.cc:492] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: No bootstrap required, opened a new log
I20260927 21:44:12.317601 19033 ts_tablet_manager.cc:1397] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 21:44:12.317703 19032 ts_tablet_manager.cc:1397] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260927 21:44:12.321780 19030 ts_tablet_manager.cc:1428] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Time spent starting tablet: real 0.024s user 0.010s sys 0.012s
I20260927 21:44:12.329012 19032 raft_consensus.cc:348] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.329674 19032 raft_consensus.cc:374] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:12.329818 19032 raft_consensus.cc:729] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf055d9c69450cb44e867599f13947, State: Initialized, Role: FOLLOWER
I20260927 21:44:12.330576 19032 consensus_queue.cc:260] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.331902 18882 heartbeater.cc:503] Master 127.0.202.190:40841 was elected leader, sending a full tablet report...
I20260927 21:44:12.332659 19032 ts_tablet_manager.cc:1428] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260927 21:44:12.335173 19033 raft_consensus.cc:348] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.335758 19033 raft_consensus.cc:374] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:12.335903 19033 raft_consensus.cc:729] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41dc5df26014106ba7b7968a834dac9, State: Initialized, Role: FOLLOWER
I20260927 21:44:12.336659 19033 consensus_queue.cc:260] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.338312 19012 heartbeater.cc:503] Master 127.0.202.190:40841 was elected leader, sending a full tablet report...
I20260927 21:44:12.339244 19033 ts_tablet_manager.cc:1428] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Time spent starting tablet: real 0.021s user 0.011s sys 0.004s
W20260927 21:44:12.434305 19013 tablet.cc:2369] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:44:12.449131 18883 tablet.cc:2369] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:12.458277 19036 raft_consensus.cc:482] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:44:12.458638 19036 raft_consensus.cc:504] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.460417 19036 leader_election.cc:288] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bf055d9c69450cb44e867599f13947 (127.0.202.130:37509), e41dc5df26014106ba7b7968a834dac9 (127.0.202.131:38805)
I20260927 21:44:12.467748 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.460808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:51781 (local address 127.0.202.130:37509)
0927 21:44:12.461006 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.461014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.461580 (+ 566us) server_negotiation.cc:407] Connection header received
0927 21:44:12.461805 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.461813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.461920 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.462016 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:12.462768 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.463739 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.465883 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.466775 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.466897 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.466957 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.467348 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.467494 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:44:12.467789 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.460808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37509 (local address 127.0.202.129:51781)
0927 21:44:12.461422 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:12.461441 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:12.461632 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:12.462127 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:12.462131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:12.462165 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:12.462622 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.462640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.463870 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.463875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.465701 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.465714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.466954 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.466962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.467139 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.467181 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:44:12.467291 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:44:12.467444 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":2,"spinlock_wait_cycles":26496,"thread_start_us":339,"threads_started":1}
I20260927 21:44:12.468329 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.461916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:46055 (local address 127.0.202.131:38805)
0927 21:44:12.462146 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.462152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.462326 (+ 174us) server_negotiation.cc:407] Connection header received
0927 21:44:12.462468 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.462473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.462556 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.462665 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:12.463271 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.464101 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.466214 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.467283 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.467440 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.467505 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.467890 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.468046 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 21:44:12.468338 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.461508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:38805 (local address 127.0.202.129:46055)
0927 21:44:12.462230 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:12.462243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:12.462353 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:12.462673 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:12.462676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:12.462711 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:12.463106 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.463118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.464278 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.464282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.465989 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:12.466002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.467513 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.467520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:12.467702 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.467735 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:44:12.467844 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:44:12.467983 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":21,"thread_start_us":386,"threads_started":1}
I20260927 21:44:12.469017 18837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7d8974e00847bf9bc7a1e8fdfe60f1" candidate_uuid: "8dba2b72159c401ba07e9a576133b87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf055d9c69450cb44e867599f13947" is_pre_election: true
I20260927 21:44:12.469226 18967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7d8974e00847bf9bc7a1e8fdfe60f1" candidate_uuid: "8dba2b72159c401ba07e9a576133b87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41dc5df26014106ba7b7968a834dac9" is_pre_election: true
I20260927 21:44:12.469597 18837 raft_consensus.cc:2454] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dba2b72159c401ba07e9a576133b87f in term 0.
I20260927 21:44:12.469732 18967 raft_consensus.cc:2454] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dba2b72159c401ba07e9a576133b87f in term 0.
I20260927 21:44:12.470290 18639 leader_election.cc:302] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf055d9c69450cb44e867599f13947, 8dba2b72159c401ba07e9a576133b87f; no voters:
I20260927 21:44:12.470660 19036 raft_consensus.cc:2781] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:44:12.470820 19036 raft_consensus.cc:482] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:44:12.470917 19036 raft_consensus.cc:3037] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:12.473279 19036 raft_consensus.cc:504] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
W20260927 21:44:12.473958 18752 tablet.cc:2369] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:12.474043 19036 leader_election.cc:288] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [CANDIDATE]: Term 1 election: Requested vote from peers 78bf055d9c69450cb44e867599f13947 (127.0.202.130:37509), e41dc5df26014106ba7b7968a834dac9 (127.0.202.131:38805)
I20260927 21:44:12.474670 18837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7d8974e00847bf9bc7a1e8fdfe60f1" candidate_uuid: "8dba2b72159c401ba07e9a576133b87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf055d9c69450cb44e867599f13947"
I20260927 21:44:12.474675 18967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7d8974e00847bf9bc7a1e8fdfe60f1" candidate_uuid: "8dba2b72159c401ba07e9a576133b87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41dc5df26014106ba7b7968a834dac9"
I20260927 21:44:12.474915 18967 raft_consensus.cc:3037] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:12.474888 18837 raft_consensus.cc:3037] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:12.477200 18967 raft_consensus.cc:2454] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dba2b72159c401ba07e9a576133b87f in term 1.
I20260927 21:44:12.477205 18837 raft_consensus.cc:2454] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dba2b72159c401ba07e9a576133b87f in term 1.
I20260927 21:44:12.477617 18639 leader_election.cc:302] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dba2b72159c401ba07e9a576133b87f, e41dc5df26014106ba7b7968a834dac9; no voters:
I20260927 21:44:12.477926 19036 raft_consensus.cc:2781] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:12.478538 19036 raft_consensus.cc:686] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 1 LEADER]: Becoming Leader. State: Replica: 8dba2b72159c401ba07e9a576133b87f, State: Running, Role: LEADER
I20260927 21:44:12.479025 19036 consensus_queue.cc:237] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } }
I20260927 21:44:12.484131 18565 catalog_manager.cc:5495] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dba2b72159c401ba07e9a576133b87f (127.0.202.129). New cstate: current_term: 1 leader_uuid: "8dba2b72159c401ba07e9a576133b87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dba2b72159c401ba07e9a576133b87f" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 } health_report { overall_health: UNKNOWN } } }
I20260927 21:44:12.510798 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.505024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.0.202.129:43035)
0927 21:44:12.505212 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.505217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.505337 (+ 120us) server_negotiation.cc:407] Connection header received
0927 21:44:12.505548 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.505552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.505637 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.505770 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:44:12.506258 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.506969 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.508404 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.508956 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.509106 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.509643 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:44:12.510211 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:44:12.510330 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.510451 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.510575 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 21:44:12.520849 18967 raft_consensus.cc:1261] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 1 FOLLOWER]: Refusing update from remote peer 8dba2b72159c401ba07e9a576133b87f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:12.520855 18837 raft_consensus.cc:1261] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 1 FOLLOWER]: Refusing update from remote peer 8dba2b72159c401ba07e9a576133b87f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:12.522176 19036 consensus_queue.cc:1035] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41dc5df26014106ba7b7968a834dac9" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:12.522692 19041 consensus_queue.cc:1035] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf055d9c69450cb44e867599f13947" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:12.537233 19044 mvcc.cc:205] Tried to move back new op lower bound from 7334074173512097792 to 7334074173357977600. Current Snapshot: MvccSnapshot[applied={T|T < 7334074173512097792}]
I20260927 21:44:12.539983 19046 mvcc.cc:205] Tried to move back new op lower bound from 7334074173512097792 to 7334074173357977600. Current Snapshot: MvccSnapshot[applied={T|T < 7334074173512097792}]
I20260927 21:44:12.551669 18564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:12.552009 18564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:12.556639 18564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f83f8012e5bd4d91a194e894dbb26c55: Sending DeleteTablet for 3 replicas of tablet df7d8974e00847bf9bc7a1e8fdfe60f1
I20260927 21:44:12.557927 18686 tablet_service.cc:1514] Processing DeleteTablet for tablet df7d8974e00847bf9bc7a1e8fdfe60f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:12 UTC) from {username='slave'} at 127.0.0.1:50234
I20260927 21:44:12.558158 18817 tablet_service.cc:1514] Processing DeleteTablet for tablet df7d8974e00847bf9bc7a1e8fdfe60f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:12 UTC) from {username='slave'} at 127.0.0.1:48948
I20260927 21:44:12.558503 18947 tablet_service.cc:1514] Processing DeleteTablet for tablet df7d8974e00847bf9bc7a1e8fdfe60f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:12 UTC) from {username='slave'} at 127.0.0.1:46528
I20260927 21:44:12.559361 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 18624
I20260927 21:44:12.573865 19055 tablet_replica.cc:330] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: stopping tablet replica
I20260927 21:44:12.574481 19055 raft_consensus.cc:2229] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:12.575129 19055 raft_consensus.cc:2258] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:12.576031 19054 tablet_replica.cc:330] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: stopping tablet replica
I20260927 21:44:12.576504 19054 raft_consensus.cc:2229] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:44:12.577206 19054 raft_consensus.cc:2258] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:12.578086 19056 tablet_replica.cc:330] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: stopping tablet replica
I20260927 21:44:12.578502 19056 raft_consensus.cc:2229] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:12.579015 19056 raft_consensus.cc:2258] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:12.580686 19055 ts_tablet_manager.cc:1905] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:12.581220 19054 ts_tablet_manager.cc:1905] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:12.585950 19056 ts_tablet_manager.cc:1905] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:12.588104 19054 ts_tablet_manager.cc:1918] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:12.588311 19054 log.cc:1196] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/df7d8974e00847bf9bc7a1e8fdfe60f1
I20260927 21:44:12.588703 19055 ts_tablet_manager.cc:1918] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:12.588871 19055 log.cc:1196] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/df7d8974e00847bf9bc7a1e8fdfe60f1
I20260927 21:44:12.588913 19054 ts_tablet_manager.cc:1939] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 8dba2b72159c401ba07e9a576133b87f: Deleting consensus metadata
I20260927 21:44:12.589478 19055 ts_tablet_manager.cc:1939] T df7d8974e00847bf9bc7a1e8fdfe60f1 P 78bf055d9c69450cb44e867599f13947: Deleting consensus metadata
I20260927 21:44:12.591270 18549 catalog_manager.cc:4841] TS 8dba2b72159c401ba07e9a576133b87f (127.0.202.129:43035): tablet df7d8974e00847bf9bc7a1e8fdfe60f1 (table test-table [id=e2246108b5bf4ccca1a9ebc3501e86ad]) successfully deleted
I20260927 21:44:12.591683 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.560184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.0.202.129:43035)
0927 21:44:12.560407 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.560413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.572030 (+ 11617us) server_negotiation.cc:407] Connection header received
0927 21:44:12.573335 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.573344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.573487 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.573575 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:12.575186 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.576604 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.579244 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.579966 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.580076 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.589102 (+ 9026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:12.589117 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:12.589140 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:12.589959 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:12.590238 (+ 279us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:12.590271 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:12.590380 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.590616 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.590742 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:44:12.591720 18549 catalog_manager.cc:4841] TS 78bf055d9c69450cb44e867599f13947 (127.0.202.130:37509): tablet df7d8974e00847bf9bc7a1e8fdfe60f1 (table test-table [id=e2246108b5bf4ccca1a9ebc3501e86ad]) successfully deleted
I20260927 21:44:12.592501 18746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:12.595084 19056 ts_tablet_manager.cc:1918] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:12.595252 19056 log.cc:1196] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/df7d8974e00847bf9bc7a1e8fdfe60f1
I20260927 21:44:12.595791 19056 ts_tablet_manager.cc:1939] T df7d8974e00847bf9bc7a1e8fdfe60f1 P e41dc5df26014106ba7b7968a834dac9: Deleting consensus metadata
I20260927 21:44:12.597622 18549 catalog_manager.cc:4841] TS e41dc5df26014106ba7b7968a834dac9 (127.0.202.131:38805): tablet df7d8974e00847bf9bc7a1e8fdfe60f1 (table test-table [id=e2246108b5bf4ccca1a9ebc3501e86ad]) successfully deleted
I20260927 21:44:12.698211 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 18624
I20260927 21:44:12.714883 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 18755
I20260927 21:44:12.722252 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.715734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.0.202.130:37509)
0927 21:44:12.715909 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.715917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.716019 (+ 102us) server_negotiation.cc:407] Connection header received
0927 21:44:12.716203 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.716207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.716394 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.716506 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:12.716918 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.717653 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.718992 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.719398 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.719534 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.720792 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:12.720802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:12.720816 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:12.721434 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:12.721495 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:12.721515 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:12.721652 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.721785 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.721897 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:44:12.723107 18877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:12.814884 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 18755
I20260927 21:44:12.834079 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 18885
I20260927 21:44:12.841454 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:12.834914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.0.202.131:38805)
0927 21:44:12.835090 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:12.835095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:12.835115 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:44:12.835232 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:12.835236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:12.835336 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:12.835467 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:12.835845 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.836786 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.838108 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:12.838552 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:12.838675 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:12.839891 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:12.839914 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:12.839938 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:12.840768 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:12.840821 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:12.840842 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:12.840947 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:12.841089 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:44:12.841188 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:44:12.842166 19007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:12.934266 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 18885
I20260927 21:44:12.953128 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 18534
I20260927 21:44:12.954265 18595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:13.070020 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 18534
2026-09-27T21:44:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2989 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:13 dist-test-slave-25dq krb5kdc[19064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:13 dist-test-slave-25dq krb5kdc[19064](info): set up 2 sockets
Sep 27 21:44:13 dist-test-slave-25dq krb5kdc[19064](info): commencing operation
krb5kdc: starting...
W20260927 21:44:14.240655 810 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:14 dist-test-slave-25dq krb5kdc[19064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:44:14.505677 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44573
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:44573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:14.862298 19077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:14.862609 19077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:14.862761 19077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:14.870169 19077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:14.870297 19077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:14.870368 19077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:14.870450 19077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:14.870509 19077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:14.880863 19077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:44573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:44573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:14.882252 19077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:14.883620 19077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:14.895900 19082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:14.895939 19085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:14.895917 19083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:14.896792 19077 server_base.cc:1034] running on GCE node
I20260927 21:44:14.897622 19077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:14.899150 19077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:14.900390 19077 hybrid_clock.cc:648] HybridClock initialized: now 1790545454900326 us; error 49 us; skew 500 ppm
I20260927 21:44:14.900743 19077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:14 dist-test-slave-25dq krb5kdc[19064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:14.903927 19077 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:44:14.905725 19077 webserver.cc:460] Webserver started at http://127.0.202.190:38147/ using document root <none> and password file <none>
I20260927 21:44:14.906442 19077 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:14.906591 19077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:14.906904 19077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:14.909941 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "5b93c721397c45f7a6b48b13ff43923c"
format_stamp: "Formatted at 2026-09-27 21:44:14 on dist-test-slave-25dq"
server_key: "382a779a9d20f701c7e18d7faefc7d4b"
server_key_iv: "effb178f2c1995e6560db553bf3a4905"
server_key_version: "encryptionkey@0"
I20260927 21:44:14.910718 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "5b93c721397c45f7a6b48b13ff43923c"
format_stamp: "Formatted at 2026-09-27 21:44:14 on dist-test-slave-25dq"
server_key: "382a779a9d20f701c7e18d7faefc7d4b"
server_key_iv: "effb178f2c1995e6560db553bf3a4905"
server_key_version: "encryptionkey@0"
I20260927 21:44:14.916119 19077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:44:14.919832 19092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:14.921355 19077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:44:14.921554 19077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "5b93c721397c45f7a6b48b13ff43923c"
format_stamp: "Formatted at 2026-09-27 21:44:14 on dist-test-slave-25dq"
server_key: "382a779a9d20f701c7e18d7faefc7d4b"
server_key_iv: "effb178f2c1995e6560db553bf3a4905"
server_key_version: "encryptionkey@0"
I20260927 21:44:14.921770 19077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:14.937734 19077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:14.942065 19077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:14.942350 19077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:14.960992 19077 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:44573
I20260927 21:44:14.961014 19143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:44573 every 8 connection(s)
I20260927 21:44:14.962412 19077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:14.966271 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:14.972570 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19077
I20260927 21:44:14.972772 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:14.973295 810 external_mini_cluster.cc:1428] Setting key 12005db0b70add2bedcba75584d65761
I20260927 21:44:14.977913 19144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c: Bootstrap starting.
I20260927 21:44:14.981124 19144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:14.982229 19144 log.cc:824] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:14.985565 19144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c: No bootstrap required, opened a new log
Sep 27 21:44:14 dist-test-slave-25dq krb5kdc[19064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:14.995217 19144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b93c721397c45f7a6b48b13ff43923c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44573 } }
I20260927 21:44:14.995743 19144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:14.995858 19144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b93c721397c45f7a6b48b13ff43923c, State: Initialized, Role: FOLLOWER
I20260927 21:44:14.996590 19144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b93c721397c45f7a6b48b13ff43923c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44573 } }
I20260927 21:44:14.996840 19144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:14.997014 19144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:14.997189 19144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:14.997742 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:14.975395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.0.202.190:44573)
0927 21:44:14.976243 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:14.976256 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:14.976356 (+ 100us) server_negotiation.cc:407] Connection header received
0927 21:44:14.978221 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:14.978246 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:14.978677 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:14.979079 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:14.980706 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:14.982182 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:14.983993 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:14.984781 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:14.984983 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:14.989689 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:14.989699 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:14.989716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:14.989758 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:14.993124 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:14.994032 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:14.994039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:14.994046 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:14.994213 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:14.994633 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:14.994636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:14.994638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:14.995176 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:14.995394 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:14.996006 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 21:44:14.996471 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":379,"threads_started":1}
I20260927 21:44:14.999650 19144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b93c721397c45f7a6b48b13ff43923c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44573 } }
I20260927 21:44:15.000137 19144 leader_election.cc:302] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b93c721397c45f7a6b48b13ff43923c; no voters:
I20260927 21:44:15.000883 19144 leader_election.cc:288] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:15.000968 19149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:15.001681 19149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [term 1 LEADER]: Becoming Leader. State: Replica: 5b93c721397c45f7a6b48b13ff43923c, State: Running, Role: LEADER
I20260927 21:44:15.002319 19149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b93c721397c45f7a6b48b13ff43923c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44573 } }
I20260927 21:44:15.003170 19144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:15.005403 19150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b93c721397c45f7a6b48b13ff43923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b93c721397c45f7a6b48b13ff43923c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44573 } } }
I20260927 21:44:15.005553 19151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b93c721397c45f7a6b48b13ff43923c. Latest consensus state: current_term: 1 leader_uuid: "5b93c721397c45f7a6b48b13ff43923c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b93c721397c45f7a6b48b13ff43923c" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 44573 } } }
I20260927 21:44:15.005962 19151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:15.005976 19150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:15.012017 19156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:15.017654 19156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:15.027241 19156 catalog_manager.cc:1347] Generated new cluster ID: 5584ee3e782142d181adbb3fde190361
I20260927 21:44:15.027365 19156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:15.038295 19156 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:15.039853 19156 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:15.047278 19156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c: Generated new TSK 0
I20260927 21:44:15.048228 19156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:44:15.116155 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44573
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:15.460585 19170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:15.460935 19170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:15.461139 19170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:15.468694 19170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:15.468884 19170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:15.469075 19170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:44:15.479372 19170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:15.480818 19170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:15.482164 19170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:15.497224 19178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:15.497716 19175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:15.497450 19176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:15.499250 19170 server_base.cc:1034] running on GCE node
I20260927 21:44:15.499969 19170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:15.501091 19170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:15.502310 19170 hybrid_clock.cc:648] HybridClock initialized: now 1790545455502228 us; error 72 us; skew 500 ppm
I20260927 21:44:15.502686 19170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:15 dist-test-slave-25dq krb5kdc[19064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:15.506942 19170 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:44:15.508975 19170 webserver.cc:460] Webserver started at http://127.0.202.129:43503/ using document root <none> and password file <none>
I20260927 21:44:15.509677 19170 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:15.509779 19170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:15.510070 19170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:15.513468 19170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "c9ead831a4d5495d96f556e2ca775ce9"
format_stamp: "Formatted at 2026-09-27 21:44:15 on dist-test-slave-25dq"
server_key: "8adbc48da3d1a7ebf47a7c7ade65ce96"
server_key_iv: "a84bc57143e62599ca386b3e5b76afbd"
server_key_version: "encryptionkey@0"
I20260927 21:44:15.514420 19170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "c9ead831a4d5495d96f556e2ca775ce9"
format_stamp: "Formatted at 2026-09-27 21:44:15 on dist-test-slave-25dq"
server_key: "8adbc48da3d1a7ebf47a7c7ade65ce96"
server_key_iv: "a84bc57143e62599ca386b3e5b76afbd"
server_key_version: "encryptionkey@0"
I20260927 21:44:15.520078 19170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:44:15.524192 19185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:15.525746 19170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:44:15.525929 19170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "c9ead831a4d5495d96f556e2ca775ce9"
format_stamp: "Formatted at 2026-09-27 21:44:15 on dist-test-slave-25dq"
server_key: "8adbc48da3d1a7ebf47a7c7ade65ce96"
server_key_iv: "a84bc57143e62599ca386b3e5b76afbd"
server_key_version: "encryptionkey@0"
I20260927 21:44:15.526132 19170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:15.554483 19170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:15.558709 19170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:15.558956 19170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:15.559975 19170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:15.561404 19170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:15.561480 19170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:15.561571 19170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:15.561614 19170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:15.596064 19170 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:34567
I20260927 21:44:15.596122 19297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:34567 every 8 connection(s)
I20260927 21:44:15.597477 19170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:44:15.605852 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19170
I20260927 21:44:15.606076 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:44:15.606689 810 external_mini_cluster.cc:1428] Setting key a0f1eea789fb8dc1de505650f44fe4bc
Sep 27 21:44:15 dist-test-slave-25dq krb5kdc[19064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:15.621945 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:15.600375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:53621 (local address 127.0.202.190:44573)
0927 21:44:15.600872 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:15.600877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:15.601921 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 21:44:15.604602 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:15.604607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:15.604696 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:15.604799 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:15.606743 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:15.607847 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:15.609733 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:15.610193 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:15.610341 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:15.614547 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:15.614553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:15.614556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:15.614593 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:15.617942 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:15.619055 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:15.619060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:15.619062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:15.619202 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:15.619757 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:15.619763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:15.619767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:15.620091 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:15.620224 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:15.621261 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0927 21:44:15.621599 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
I20260927 21:44:15.623261 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:15.600879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44573 (local address 127.0.202.129:53621)
0927 21:44:15.601698 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:15.601744 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:15.604131 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:15.605039 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:15.605047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:15.605674 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:15.606528 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:15.606550 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:15.608034 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:15.608045 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:15.609559 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:15.609574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:15.610481 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:15.610489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:15.610743 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:15.611592 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:15.611623 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:15.614268 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:15.618137 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:15.618145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:15.618160 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:15.618899 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:15.619321 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:15.619325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:15.619328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:15.619599 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:15.620228 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:15.620239 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:15.620782 (+ 543us) client_negotiation.cc:769] Sending connection context
0927 21:44:15.621126 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 21:44:15.621748 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":335,"threads_started":1}
I20260927 21:44:15.625017 19298 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44573
I20260927 21:44:15.625383 19298 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:15.626235 19298 heartbeater.cc:511] Master 127.0.202.190:44573 requested a full tablet report, sending...
I20260927 21:44:15.628468 19109 ts_manager.cc:194] Registered new tserver with Master: c9ead831a4d5495d96f556e2ca775ce9 (127.0.202.129:34567)
I20260927 21:44:15.630101 19109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:53621
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:44:15.668995 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44573
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:16.024964 19304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:16.025306 19304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:16.025413 19304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:16.033465 19304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:16.033615 19304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:16.033741 19304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:44:16.043856 19304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:16.045217 19304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:16.046636 19304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:16.061051 19309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:16.061098 19312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:16.061017 19310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:16.062028 19304 server_base.cc:1034] running on GCE node
I20260927 21:44:16.062701 19304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:16.063755 19304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:16.065011 19304 hybrid_clock.cc:648] HybridClock initialized: now 1790545456064917 us; error 55 us; skew 500 ppm
I20260927 21:44:16.065343 19304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:16 dist-test-slave-25dq krb5kdc[19064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:16.068589 19304 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:44:16.070127 19304 webserver.cc:460] Webserver started at http://127.0.202.130:43887/ using document root <none> and password file <none>
I20260927 21:44:16.070845 19304 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:16.070992 19304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:16.071303 19304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:16.074321 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "4476beb9a82845c09f1bacb1efa7522b"
format_stamp: "Formatted at 2026-09-27 21:44:16 on dist-test-slave-25dq"
server_key: "7726660e90f02069ace434efca4f54e7"
server_key_iv: "b879ff7d1304abd6484caef9fc070cf0"
server_key_version: "encryptionkey@0"
I20260927 21:44:16.075146 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "4476beb9a82845c09f1bacb1efa7522b"
format_stamp: "Formatted at 2026-09-27 21:44:16 on dist-test-slave-25dq"
server_key: "7726660e90f02069ace434efca4f54e7"
server_key_iv: "b879ff7d1304abd6484caef9fc070cf0"
server_key_version: "encryptionkey@0"
I20260927 21:44:16.080826 19304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:44:16.084797 19319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:16.086279 19304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:44:16.086477 19304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "4476beb9a82845c09f1bacb1efa7522b"
format_stamp: "Formatted at 2026-09-27 21:44:16 on dist-test-slave-25dq"
server_key: "7726660e90f02069ace434efca4f54e7"
server_key_iv: "b879ff7d1304abd6484caef9fc070cf0"
server_key_version: "encryptionkey@0"
I20260927 21:44:16.086710 19304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:16.128329 19304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:16.133260 19304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:16.133589 19304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:16.134588 19304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:16.135984 19304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:16.136096 19304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:16.136245 19304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:16.136335 19304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:16.169759 19304 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:37041
I20260927 21:44:16.169812 19431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:37041 every 8 connection(s)
I20260927 21:44:16.171115 19304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:44:16.175665 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19304
I20260927 21:44:16.175866 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:44:16.176501 810 external_mini_cluster.cc:1428] Setting key 5d0c4c24bada0a4386ce1ec5e0657ecd
Sep 27 21:44:16 dist-test-slave-25dq krb5kdc[19064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:16.197103 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.173908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:47151 (local address 127.0.202.190:44573)
0927 21:44:16.174450 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.174460 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.175581 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 21:44:16.178093 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.178098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.178184 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.178298 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:16.180426 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.181147 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.183663 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.184149 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.184249 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.188859 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:16.188869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:16.188876 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:16.188934 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:16.193157 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:16.193998 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:16.194007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:16.194014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:16.194191 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:16.194675 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:16.194683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:16.194690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:16.195147 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:16.195318 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.196208 (+ 890us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.196524 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":227,"threads_started":1}
I20260927 21:44:16.197983 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.174566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44573 (local address 127.0.202.130:47151)
0927 21:44:16.175307 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:16.175371 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:16.177755 (+ 2384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:16.178592 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:16.178601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:16.179351 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:16.180217 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.180246 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.181317 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.181325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.183494 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.183512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.184433 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.184443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.184801 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.185605 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:16.185639 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:16.188587 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:16.193327 (+ 4740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:16.193334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:16.193352 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:16.193822 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:16.194316 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:16.194319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:16.194322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:16.194530 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:16.195319 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:16.195328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:16.195761 (+ 433us) client_negotiation.cc:769] Sending connection context
0927 21:44:16.196038 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 21:44:16.196635 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":278,"threads_started":1}
I20260927 21:44:16.199618 19432 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44573
I20260927 21:44:16.199970 19432 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:16.200814 19432 heartbeater.cc:511] Master 127.0.202.190:44573 requested a full tablet report, sending...
I20260927 21:44:16.202402 19109 ts_manager.cc:194] Registered new tserver with Master: 4476beb9a82845c09f1bacb1efa7522b (127.0.202.130:37041)
I20260927 21:44:16.203351 19109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:47151
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:44:16.241051 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44573
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:16.580502 19438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:16.580799 19438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:16.580896 19438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:16.588505 19438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:16.588626 19438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:16.588742 19438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:44:16.601938 19438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:44573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:16.603438 19438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:16.604911 19438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:16.619027 19443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:16.618973 19446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:16.619065 19444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:16.619737 19438 server_base.cc:1034] running on GCE node
I20260927 21:44:16.620510 19438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:16.621534 19438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:16.622769 19438 hybrid_clock.cc:648] HybridClock initialized: now 1790545456622722 us; error 46 us; skew 500 ppm
I20260927 21:44:16.623111 19438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:16 dist-test-slave-25dq krb5kdc[19064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:16.627058 19438 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:44:16.628842 19438 webserver.cc:460] Webserver started at http://127.0.202.131:35141/ using document root <none> and password file <none>
I20260927 21:44:16.629886 19438 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:16.630044 19438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:16.630360 19438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:16.633204 19298 heartbeater.cc:503] Master 127.0.202.190:44573 was elected leader, sending a full tablet report...
I20260927 21:44:16.633658 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "7ac42f92d7be49d1aee8757300c3adf4"
format_stamp: "Formatted at 2026-09-27 21:44:16 on dist-test-slave-25dq"
server_key: "10f6f726afc66d6c6c222e0f4f13ae68"
server_key_iv: "d947d0998a32bd6849b1f2cc64064adc"
server_key_version: "encryptionkey@0"
I20260927 21:44:16.635358 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "7ac42f92d7be49d1aee8757300c3adf4"
format_stamp: "Formatted at 2026-09-27 21:44:16 on dist-test-slave-25dq"
server_key: "10f6f726afc66d6c6c222e0f4f13ae68"
server_key_iv: "d947d0998a32bd6849b1f2cc64064adc"
server_key_version: "encryptionkey@0"
I20260927 21:44:16.641268 19438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:44:16.644935 19453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:16.646574 19438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:16.646780 19438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "7ac42f92d7be49d1aee8757300c3adf4"
format_stamp: "Formatted at 2026-09-27 21:44:16 on dist-test-slave-25dq"
server_key: "10f6f726afc66d6c6c222e0f4f13ae68"
server_key_iv: "d947d0998a32bd6849b1f2cc64064adc"
server_key_version: "encryptionkey@0"
I20260927 21:44:16.647008 19438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:16.679827 19438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:16.684525 19438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:16.684880 19438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:16.685869 19438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:16.687212 19438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:16.687322 19438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:16.687464 19438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:16.687532 19438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:16.723107 19438 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:42477
I20260927 21:44:16.723147 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:42477 every 8 connection(s)
I20260927 21:44:16.724561 19438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:44:16.727301 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19438
I20260927 21:44:16.727496 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:44:16.728036 810 external_mini_cluster.cc:1428] Setting key 3adcdd0c85ec47464608042565398442
Sep 27 21:44:16 dist-test-slave-25dq krb5kdc[19064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:16.746083 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.727530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:37863 (local address 127.0.202.190:44573)
0927 21:44:16.728086 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.728112 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.729614 (+ 1502us) server_negotiation.cc:407] Connection header received
0927 21:44:16.731429 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.731435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.731541 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.731664 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:16.733330 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.734123 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.735713 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.736144 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.736283 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.739772 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:16.739777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:16.739780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:16.739814 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:16.742976 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:16.743711 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:16.743717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:16.743725 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:16.743840 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:16.744317 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:16.744322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:16.744329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:16.744607 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:16.744769 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.745483 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.745705 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":300,"threads_started":1}
I20260927 21:44:16.747107 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.728185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:44573 (local address 127.0.202.131:37863)
0927 21:44:16.729098 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:16.729145 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:16.731100 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:16.731920 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:16.731929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:16.732543 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:16.733158 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.733177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.734291 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.734295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.735561 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.735575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.736408 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.736415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.736651 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.737287 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:16.737314 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:16.739554 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:16.743105 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:16.743114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:16.743138 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:16.743584 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:16.743990 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:16.743994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:16.743996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:16.744179 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:16.744738 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:16.744746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:16.745105 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 21:44:16.745371 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 21:44:16.745905 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":388,"threads_started":1}
I20260927 21:44:16.748991 19566 heartbeater.cc:347] Connected to a master server at 127.0.202.190:44573
I20260927 21:44:16.749342 19566 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:16.750106 19566 heartbeater.cc:511] Master 127.0.202.190:44573 requested a full tablet report, sending...
I20260927 21:44:16.751674 19109 ts_manager.cc:194] Registered new tserver with Master: 7ac42f92d7be49d1aee8757300c3adf4 (127.0.202.131:42477)
I20260927 21:44:16.752700 19109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:37863
I20260927 21:44:16.756160 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:44:16.775079 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.760151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.0.202.190:44573)
0927 21:44:16.760376 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.760383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.760911 (+ 528us) server_negotiation.cc:407] Connection header received
0927 21:44:16.761027 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.761031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.761095 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.761190 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:16.762651 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.763391 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.764948 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.765475 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.765604 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.768893 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:16.768899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:16.768902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:16.768943 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:16.771736 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:16.772774 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:16.772779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:16.772781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:16.772959 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:16.773519 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:16.773523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:16.773526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:16.773856 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:16.774071 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.774642 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.774807 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:44:16.779666 19109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:44:16.787155 19109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:44:16.810593 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.800272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.0.202.129:34567)
0927 21:44:16.800956 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.800968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.801530 (+ 562us) server_negotiation.cc:407] Connection header received
0927 21:44:16.801826 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.801832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.802129 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.802332 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:16.804144 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.805869 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.807677 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.809194 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.809322 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.809598 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.810065 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.810234 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":289,"threads_started":1}
I20260927 21:44:16.810653 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.800103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34567 (local address 127.0.0.1:50662)
0927 21:44:16.801346 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:16.801410 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:16.801624 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:16.802705 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:16.802717 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:16.802763 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:16.803310 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.803328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.805997 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.806005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.807423 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.807435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.809464 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.809471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.809802 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.809844 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:44:16.810019 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 21:44:16.810218 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":55,"thread_start_us":806,"threads_started":1}
I20260927 21:44:16.813441 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.801836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37041 (local address 127.0.0.1:34600)
0927 21:44:16.802395 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:16.802412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:16.802539 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:16.805137 (+ 2598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:16.805142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:16.805168 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:16.805605 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.805620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.807664 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.807672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.810404 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.810419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.812462 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.812470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.812648 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.812685 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:44:16.812756 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:44:16.812916 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":207,"thread_start_us":295,"threads_started":1}
I20260927 21:44:16.814152 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.802097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.0.202.130:37041)
0927 21:44:16.804639 (+ 2542us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.804649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.804677 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:44:16.804763 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.804769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.804939 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.805135 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:16.805790 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.807479 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.810621 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.812188 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.813168 (+ 980us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.813382 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.813468 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.813780 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2381,"thread_start_us":2245,"threads_started":1}
I20260927 21:44:16.814249 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.802489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42477 (local address 127.0.0.1:33922)
0927 21:44:16.803101 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:16.803118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:16.803239 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:16.805421 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:16.805424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:16.805447 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:16.805841 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.805854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.807776 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.807782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.810421 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.810430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.812886 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.812892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.813086 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.813119 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:44:16.813183 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:44:16.813314 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":171,"thread_start_us":861,"threads_started":1}
I20260927 21:44:16.815678 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.803079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.0.202.131:42477)
0927 21:44:16.804053 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.804061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.804094 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:44:16.804629 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.804639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.804924 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.805135 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:16.806066 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.807616 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.810596 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.812650 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.814870 (+ 2220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.815063 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.815162 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.815335 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":655,"threads_started":1}
I20260927 21:44:16.815272 19233 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 (DEFAULT_TABLE table=test-table [id=103d7be3b5c347fbaf42dae28de33770]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:16.816679 19233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:16.817714 19367 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 (DEFAULT_TABLE table=test-table [id=103d7be3b5c347fbaf42dae28de33770]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:16.818993 19367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:16.819278 19501 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 (DEFAULT_TABLE table=test-table [id=103d7be3b5c347fbaf42dae28de33770]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:16.820740 19501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:16.831568 19585 tablet_bootstrap.cc:492] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Bootstrap starting.
I20260927 21:44:16.836656 19585 tablet_bootstrap.cc:654] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:16.837175 19586 tablet_bootstrap.cc:492] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Bootstrap starting.
I20260927 21:44:16.838431 19585 log.cc:824] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:16.839003 19587 tablet_bootstrap.cc:492] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Bootstrap starting.
I20260927 21:44:16.841323 19586 tablet_bootstrap.cc:654] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:16.842530 19587 tablet_bootstrap.cc:654] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:16.842573 19585 tablet_bootstrap.cc:492] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: No bootstrap required, opened a new log
I20260927 21:44:16.842747 19586 log.cc:824] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:16.842854 19585 ts_tablet_manager.cc:1397] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260927 21:44:16.843761 19587 log.cc:824] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:16.846565 19587 tablet_bootstrap.cc:492] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: No bootstrap required, opened a new log
I20260927 21:44:16.846681 19586 tablet_bootstrap.cc:492] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: No bootstrap required, opened a new log
I20260927 21:44:16.846884 19587 ts_tablet_manager.cc:1397] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 21:44:16.846968 19586 ts_tablet_manager.cc:1397] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 21:44:16.853736 19585 raft_consensus.cc:348] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.854239 19585 raft_consensus.cc:374] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:16.854328 19585 raft_consensus.cc:729] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4476beb9a82845c09f1bacb1efa7522b, State: Initialized, Role: FOLLOWER
I20260927 21:44:16.854980 19585 consensus_queue.cc:260] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.856216 19432 heartbeater.cc:503] Master 127.0.202.190:44573 was elected leader, sending a full tablet report...
I20260927 21:44:16.856371 19587 raft_consensus.cc:348] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.856840 19587 raft_consensus.cc:374] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:16.856940 19587 raft_consensus.cc:729] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac42f92d7be49d1aee8757300c3adf4, State: Initialized, Role: FOLLOWER
I20260927 21:44:16.857328 19585 ts_tablet_manager.cc:1428] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260927 21:44:16.857409 19586 raft_consensus.cc:348] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.857826 19586 raft_consensus.cc:374] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:16.857685 19587 consensus_queue.cc:260] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.857930 19586 raft_consensus.cc:729] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ead831a4d5495d96f556e2ca775ce9, State: Initialized, Role: FOLLOWER
I20260927 21:44:16.858979 19566 heartbeater.cc:503] Master 127.0.202.190:44573 was elected leader, sending a full tablet report...
I20260927 21:44:16.859634 19587 ts_tablet_manager.cc:1428] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260927 21:44:16.858603 19586 consensus_queue.cc:260] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.862839 19586 ts_tablet_manager.cc:1428] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260927 21:44:16.923278 19433 tablet.cc:2369] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:16.958796 19591 raft_consensus.cc:482] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:44:16.959158 19591 raft_consensus.cc:504] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.961077 19591 leader_election.cc:288] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9ead831a4d5495d96f556e2ca775ce9 (127.0.202.129:34567), 7ac42f92d7be49d1aee8757300c3adf4 (127.0.202.131:42477)
I20260927 21:44:16.968978 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.961507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:34707 (local address 127.0.202.129:34567)
0927 21:44:16.961769 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.961779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.962246 (+ 467us) server_negotiation.cc:407] Connection header received
0927 21:44:16.962593 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.962599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.962669 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.962725 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:16.963500 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.964545 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.966703 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.967966 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.968129 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.968205 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.968624 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.968750 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:44:16.969434 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.962222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:42477 (local address 127.0.202.130:58425)
0927 21:44:16.963272 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:16.963290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:16.963446 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:16.963790 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:16.963795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:16.963820 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:16.964146 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.964157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.965611 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.965616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.967282 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.967294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.968590 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.968597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.968784 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.968820 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:44:16.968907 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:44:16.969039 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":1,"thread_start_us":667,"threads_started":1}
I20260927 21:44:16.969815 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.962476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:58425 (local address 127.0.202.131:42477)
0927 21:44:16.962726 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:16.962731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:16.963397 (+ 666us) server_negotiation.cc:407] Connection header received
0927 21:44:16.963594 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:16.963601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:16.963673 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:16.963750 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:16.964351 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.965462 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.967440 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.968355 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.968524 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.968613 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:16.968954 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 21:44:16.969362 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 21:44:16.970029 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:16.961428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34567 (local address 127.0.202.130:34707)
0927 21:44:16.962135 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:16.962159 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:16.962434 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:16.962912 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:16.962917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:16.962942 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:16.963375 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.963385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.964710 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.964715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.966533 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:16.966548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:16.968191 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:16.968197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:16.968414 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:16.968456 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:44:16.969252 (+ 796us) client_negotiation.cc:240] Negotiation successful
0927 21:44:16.969391 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":382,"threads_started":1}
I20260927 21:44:16.970919 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7fe6d8e39b4346a420d3de0f3a3d67" candidate_uuid: "4476beb9a82845c09f1bacb1efa7522b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac42f92d7be49d1aee8757300c3adf4" is_pre_election: true
I20260927 21:44:16.970957 19253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7fe6d8e39b4346a420d3de0f3a3d67" candidate_uuid: "4476beb9a82845c09f1bacb1efa7522b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ead831a4d5495d96f556e2ca775ce9" is_pre_election: true
I20260927 21:44:16.971530 19521 raft_consensus.cc:2454] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4476beb9a82845c09f1bacb1efa7522b in term 0.
I20260927 21:44:16.971530 19253 raft_consensus.cc:2454] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4476beb9a82845c09f1bacb1efa7522b in term 0.
I20260927 21:44:16.972389 19320 leader_election.cc:302] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4476beb9a82845c09f1bacb1efa7522b, c9ead831a4d5495d96f556e2ca775ce9; no voters:
I20260927 21:44:16.972874 19591 raft_consensus.cc:2781] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:44:16.973029 19591 raft_consensus.cc:482] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:44:16.973150 19591 raft_consensus.cc:3037] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:16.975569 19591 raft_consensus.cc:504] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
W20260927 21:44:16.976332 19567 tablet.cc:2369] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:16.976411 19591 leader_election.cc:288] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [CANDIDATE]: Term 1 election: Requested vote from peers c9ead831a4d5495d96f556e2ca775ce9 (127.0.202.129:34567), 7ac42f92d7be49d1aee8757300c3adf4 (127.0.202.131:42477)
I20260927 21:44:16.977020 19253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7fe6d8e39b4346a420d3de0f3a3d67" candidate_uuid: "4476beb9a82845c09f1bacb1efa7522b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ead831a4d5495d96f556e2ca775ce9"
I20260927 21:44:16.977280 19253 raft_consensus.cc:3037] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:16.977272 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7fe6d8e39b4346a420d3de0f3a3d67" candidate_uuid: "4476beb9a82845c09f1bacb1efa7522b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac42f92d7be49d1aee8757300c3adf4"
I20260927 21:44:16.977481 19521 raft_consensus.cc:3037] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:16.979791 19521 raft_consensus.cc:2454] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4476beb9a82845c09f1bacb1efa7522b in term 1.
I20260927 21:44:16.979800 19253 raft_consensus.cc:2454] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4476beb9a82845c09f1bacb1efa7522b in term 1.
I20260927 21:44:16.980268 19320 leader_election.cc:302] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4476beb9a82845c09f1bacb1efa7522b, c9ead831a4d5495d96f556e2ca775ce9; no voters:
I20260927 21:44:16.980662 19591 raft_consensus.cc:2781] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:16.981166 19591 raft_consensus.cc:686] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 1 LEADER]: Becoming Leader. State: Replica: 4476beb9a82845c09f1bacb1efa7522b, State: Running, Role: LEADER
I20260927 21:44:16.981689 19591 consensus_queue.cc:237] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } }
I20260927 21:44:16.986918 19107 catalog_manager.cc:5495] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4476beb9a82845c09f1bacb1efa7522b (127.0.202.130). New cstate: current_term: 1 leader_uuid: "4476beb9a82845c09f1bacb1efa7522b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4476beb9a82845c09f1bacb1efa7522b" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 } health_report { overall_health: UNKNOWN } } }
I20260927 21:44:17.010242 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:17.003908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.202.130:37041)
0927 21:44:17.004125 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:17.004132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:17.004239 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:44:17.004537 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:17.004542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:17.004684 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:17.004862 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:44:17.005355 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.006155 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.007725 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.008182 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.008381 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:17.009028 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:44:17.009569 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:44:17.009772 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:17.009915 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 21:44:17.009978 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:44:17.019305 19253 raft_consensus.cc:1261] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 1 FOLLOWER]: Refusing update from remote peer 4476beb9a82845c09f1bacb1efa7522b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:17.019309 19521 raft_consensus.cc:1261] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 1 FOLLOWER]: Refusing update from remote peer 4476beb9a82845c09f1bacb1efa7522b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:17.020392 19596 consensus_queue.cc:1035] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ead831a4d5495d96f556e2ca775ce9" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:17.020813 19591 consensus_queue.cc:1035] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac42f92d7be49d1aee8757300c3adf4" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 42477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:17.034194 19601 mvcc.cc:205] Tried to move back new op lower bound from 7334074191940132864 to 7334074191801286656. Current Snapshot: MvccSnapshot[applied={T|T < 7334074191940132864}]
I20260927 21:44:17.048213 19108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:17.048651 19108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:17.053193 19108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b93c721397c45f7a6b48b13ff43923c: Sending DeleteTablet for 3 replicas of tablet 3e7fe6d8e39b4346a420d3de0f3a3d67
I20260927 21:44:17.054754 19233 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:17 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:50662
I20260927 21:44:17.054853 19501 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:17 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:33922
I20260927 21:44:17.056006 19367 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:17 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:34600
I20260927 21:44:17.057540 19611 tablet_replica.cc:330] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: stopping tablet replica
I20260927 21:44:17.058108 19611 raft_consensus.cc:2229] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:44:17.059175 19609 tablet_replica.cc:330] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: stopping tablet replica
I20260927 21:44:17.059206 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19170
I20260927 21:44:17.059643 19609 raft_consensus.cc:2229] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:17.060171 19609 raft_consensus.cc:2258] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:17.060503 19610 tablet_replica.cc:330] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: stopping tablet replica
I20260927 21:44:17.060974 19610 raft_consensus.cc:2229] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:17.061547 19610 raft_consensus.cc:2258] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:17.061690 19611 raft_consensus.cc:2258] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:17.063503 19609 ts_tablet_manager.cc:1905] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:17.063503 19610 ts_tablet_manager.cc:1905] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:17.070745 19609 ts_tablet_manager.cc:1918] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:17.070745 19610 ts_tablet_manager.cc:1918] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:17.070935 19609 log.cc:1196] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/3e7fe6d8e39b4346a420d3de0f3a3d67
I20260927 21:44:17.071117 19610 log.cc:1196] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/3e7fe6d8e39b4346a420d3de0f3a3d67
I20260927 21:44:17.071507 19609 ts_tablet_manager.cc:1939] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 7ac42f92d7be49d1aee8757300c3adf4: Deleting consensus metadata
I20260927 21:44:17.071854 19610 ts_tablet_manager.cc:1939] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P c9ead831a4d5495d96f556e2ca775ce9: Deleting consensus metadata
Sep 27 21:44:17 dist-test-slave-25dq krb5kdc[19064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:44:17.073801 19093 catalog_manager.cc:4841] TS c9ead831a4d5495d96f556e2ca775ce9 (127.0.202.129:34567): tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 (table test-table [id=103d7be3b5c347fbaf42dae28de33770]) successfully deleted
I20260927 21:44:17.074371 19611 ts_tablet_manager.cc:1905] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:17.075510 19093 catalog_manager.cc:4841] TS 7ac42f92d7be49d1aee8757300c3adf4 (127.0.202.131:42477): tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 (table test-table [id=103d7be3b5c347fbaf42dae28de33770]) successfully deleted
I20260927 21:44:17.081089 19611 ts_tablet_manager.cc:1918] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:17.081322 19611 log.cc:1196] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/3e7fe6d8e39b4346a420d3de0f3a3d67
I20260927 21:44:17.081770 19611 ts_tablet_manager.cc:1939] T 3e7fe6d8e39b4346a420d3de0f3a3d67 P 4476beb9a82845c09f1bacb1efa7522b: Deleting consensus metadata
I20260927 21:44:17.083074 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:17.061180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.0.202.129:34567)
0927 21:44:17.061439 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:17.061449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:17.061931 (+ 482us) server_negotiation.cc:407] Connection header received
0927 21:44:17.062242 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:17.062249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:17.062338 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:17.062445 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:17.064494 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.065816 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.067998 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.068844 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.068983 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:17.075318 (+ 6335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:17.075337 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:17.075358 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:17.075426 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:17.079585 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:17.080546 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:17.080557 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:17.080560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:17.080728 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:17.081200 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:17.081208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:17.081213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:17.081726 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:17.082032 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:17.082447 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 21:44:17.082682 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 21:44:17.083739 19093 catalog_manager.cc:4841] TS 4476beb9a82845c09f1bacb1efa7522b (127.0.202.130:37041): tablet 3e7fe6d8e39b4346a420d3de0f3a3d67 (table test-table [id=103d7be3b5c347fbaf42dae28de33770]) successfully deleted
I20260927 21:44:17.083932 19293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:17.205484 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19170
I20260927 21:44:17.231884 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19304
Sep 27 21:44:17 dist-test-slave-25dq krb5kdc[19064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:44:17.251247 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:17.232767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.0.202.130:37041)
0927 21:44:17.232947 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:17.232952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:17.233006 (+ 54us) server_negotiation.cc:407] Connection header received
0927 21:44:17.233201 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:17.233205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:17.233280 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:17.233370 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:17.235106 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.235962 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.238219 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.238721 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.238820 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:17.244749 (+ 5929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:17.244765 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:17.244781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:17.244827 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:17.248086 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:17.249019 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:17.249027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:17.249030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:17.249181 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:17.249614 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:17.249617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:17.249619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:17.250028 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:17.250266 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:17.250675 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 21:44:17.250891 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:44:17.252115 19427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:17.367393 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19304
I20260927 21:44:17.387110 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19438
Sep 27 21:44:17 dist-test-slave-25dq krb5kdc[19064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:44:17.402650 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:17.387994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.0.202.131:42477)
0927 21:44:17.388206 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:17.388212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:17.388232 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:44:17.388432 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:17.388437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:17.388536 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:17.388650 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:17.390015 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.390740 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.392022 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:17.392515 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:17.392654 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:17.397177 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:17.397195 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:17.397212 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:17.397248 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:17.399856 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:17.400606 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:17.400621 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:17.400623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:17.400767 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:17.401146 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:17.401151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:17.401154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:17.401493 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:17.401799 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:17.402079 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 21:44:17.402313 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:44:17.403420 19561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:17.495574 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19438
I20260927 21:44:17.512429 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19077
I20260927 21:44:17.513834 19139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:17.628594 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19077
2026-09-27T21:44:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4557 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T21:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:17Z Disabled control of system clock
I20260927 21:44:17.706638 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:39363
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:39363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:44:18.046881 19620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:18.047247 19620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:18.047437 19620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:18.055174 19620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:18.055315 19620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:18.055435 19620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:18.055554 19620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:18.055619 19620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:18.065557 19620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:39363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:39363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:18.066982 19620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:18.068972 19620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:18.082127 19625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:18.082191 19628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:18.082098 19626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:18.084079 19620 server_base.cc:1034] running on GCE node
I20260927 21:44:18.085114 19620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:18.086661 19620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:18.087909 19620 hybrid_clock.cc:648] HybridClock initialized: now 1790545458087857 us; error 39 us; skew 500 ppm
I20260927 21:44:18.088441 19620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:18.100117 19620 webserver.cc:460] Webserver started at http://127.0.202.190:35455/ using document root <none> and password file <none>
I20260927 21:44:18.100934 19620 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:18.101064 19620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:18.101409 19620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:18.104437 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "d4d102a5b2b848fe968ff40ec5487832"
format_stamp: "Formatted at 2026-09-27 21:44:18 on dist-test-slave-25dq"
server_key: "b4e7b0aafb4f9abe66346c11b41d9df6"
server_key_iv: "a9234ad061783cf0540ef3292449acc4"
server_key_version: "encryptionkey@0"
I20260927 21:44:18.105266 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "d4d102a5b2b848fe968ff40ec5487832"
format_stamp: "Formatted at 2026-09-27 21:44:18 on dist-test-slave-25dq"
server_key: "b4e7b0aafb4f9abe66346c11b41d9df6"
server_key_iv: "a9234ad061783cf0540ef3292449acc4"
server_key_version: "encryptionkey@0"
I20260927 21:44:18.110724 19620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:44:18.114924 19634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:18.116603 19620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:44:18.116808 19620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "d4d102a5b2b848fe968ff40ec5487832"
format_stamp: "Formatted at 2026-09-27 21:44:18 on dist-test-slave-25dq"
server_key: "b4e7b0aafb4f9abe66346c11b41d9df6"
server_key_iv: "a9234ad061783cf0540ef3292449acc4"
server_key_version: "encryptionkey@0"
I20260927 21:44:18.117045 19620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:18.156502 19620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:18.157521 19620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:18.157884 19620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:18.179476 19620 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:39363
I20260927 21:44:18.179530 19685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:39363 every 8 connection(s)
I20260927 21:44:18.181032 19620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:18.182518 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19620
I20260927 21:44:18.182700 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:18.183151 810 external_mini_cluster.cc:1428] Setting key 9ecd9a80d165b0944c1e463b9e37b7dc
I20260927 21:44:18.185616 19686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:18.198258 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:18.185190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.202.190:39363)
0927 21:44:18.186221 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:18.186233 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:18.186275 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:44:18.187970 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:18.187992 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:18.188532 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:18.188879 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:18.189575 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:18.190991 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:18.192917 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:18.193581 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:18.193715 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:18.195112 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:18.195122 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:18.195143 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:18.195926 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:18.196044 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:18.196071 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:18.196202 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:18.196528 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 21:44:18.196944 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":382,"threads_started":1}
I20260927 21:44:18.199489 19686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832: Bootstrap starting.
I20260927 21:44:18.202694 19686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:18.203713 19686 log.cc:824] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:18.206835 19686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832: No bootstrap required, opened a new log
I20260927 21:44:18.214767 19686 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d102a5b2b848fe968ff40ec5487832" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39363 } }
I20260927 21:44:18.215188 19686 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:18.215301 19686 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d102a5b2b848fe968ff40ec5487832, State: Initialized, Role: FOLLOWER
I20260927 21:44:18.215960 19686 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d102a5b2b848fe968ff40ec5487832" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39363 } }
I20260927 21:44:18.216202 19686 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:18.216372 19686 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:18.216529 19686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:18.219094 19686 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d102a5b2b848fe968ff40ec5487832" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39363 } }
I20260927 21:44:18.219663 19686 leader_election.cc:302] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4d102a5b2b848fe968ff40ec5487832; no voters:
I20260927 21:44:18.220319 19686 leader_election.cc:288] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:18.220602 19691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:18.221487 19691 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [term 1 LEADER]: Becoming Leader. State: Replica: d4d102a5b2b848fe968ff40ec5487832, State: Running, Role: LEADER
I20260927 21:44:18.222105 19691 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d102a5b2b848fe968ff40ec5487832" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39363 } }
I20260927 21:44:18.222775 19686 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:18.225148 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4d102a5b2b848fe968ff40ec5487832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d102a5b2b848fe968ff40ec5487832" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39363 } } }
I20260927 21:44:18.225656 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:18.225332 19693 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d102a5b2b848fe968ff40ec5487832. Latest consensus state: current_term: 1 leader_uuid: "d4d102a5b2b848fe968ff40ec5487832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d102a5b2b848fe968ff40ec5487832" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 39363 } } }
I20260927 21:44:18.225912 19693 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832 [sys.catalog]: This master's current role is: LEADER
W20260927 21:44:18.238826 19707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:44:18.238957 19707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:44:18.239105 19699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:18.241662 19699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:18.251402 19699 catalog_manager.cc:1347] Generated new cluster ID: d01f881cc791407ab5341939ce65f66d
I20260927 21:44:18.251550 19699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:18.264387 19699 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:18.265729 19699 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:18.278931 19699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832: Generated new TSK 0
I20260927 21:44:18.279918 19699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:18.285358 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:39363
--builtin_ntp_servers=127.0.202.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:44:18.660816 19710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:18.661123 19710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:18.661218 19710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:18.668955 19710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:18.669070 19710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:18.669190 19710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:44:18.678560 19710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:39363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:18.679819 19710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:18.681660 19710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:18.696106 19715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:18.697365 19710 server_base.cc:1034] running on GCE node
W20260927 21:44:18.699239 19718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:18.699265 19716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:18.700155 19710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:18.701293 19710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:18.702541 19710 hybrid_clock.cc:648] HybridClock initialized: now 1790545458702464 us; error 64 us; skew 500 ppm
I20260927 21:44:18.702919 19710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:18.705588 19710 webserver.cc:460] Webserver started at http://127.0.202.129:37441/ using document root <none> and password file <none>
I20260927 21:44:18.706370 19710 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:18.706494 19710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:18.706859 19710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:18.710116 19710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "f7868538faf34d09b440cca916349d28"
format_stamp: "Formatted at 2026-09-27 21:44:18 on dist-test-slave-25dq"
server_key: "684649843bb3095656df407122205d31"
server_key_iv: "5ed2d8a4e1b068b02ea77210388b451d"
server_key_version: "encryptionkey@0"
I20260927 21:44:18.710986 19710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "f7868538faf34d09b440cca916349d28"
format_stamp: "Formatted at 2026-09-27 21:44:18 on dist-test-slave-25dq"
server_key: "684649843bb3095656df407122205d31"
server_key_iv: "5ed2d8a4e1b068b02ea77210388b451d"
server_key_version: "encryptionkey@0"
I20260927 21:44:18.717026 19710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 21:44:18.721433 19724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:18.723318 19710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:44:18.723515 19710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "f7868538faf34d09b440cca916349d28"
format_stamp: "Formatted at 2026-09-27 21:44:18 on dist-test-slave-25dq"
server_key: "684649843bb3095656df407122205d31"
server_key_iv: "5ed2d8a4e1b068b02ea77210388b451d"
server_key_version: "encryptionkey@0"
I20260927 21:44:18.723815 19710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:18.759382 19710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:18.760440 19710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:18.760779 19710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:18.761900 19710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:18.763573 19710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:18.763711 19710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:18.763840 19710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:18.763943 19710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:18.800585 19710 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:41985
I20260927 21:44:18.800678 19836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:41985 every 8 connection(s)
I20260927 21:44:18.802076 19710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:44:18.808687 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19710
I20260927 21:44:18.808887 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:44:18.809309 810 external_mini_cluster.cc:1428] Setting key 426c63ae1199237c7cf56a5b080a771b
I20260927 21:44:18.812950 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:39363
--builtin_ntp_servers=127.0.202.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:44:18.819576 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:18.805263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:39363 (local address 127.0.202.129:34409)
0927 21:44:18.806205 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:18.806255 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:18.808055 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:18.808909 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:18.808923 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:18.809067 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:18.809749 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:18.809794 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:18.811530 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:18.811536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:18.813886 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:18.813914 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:18.815325 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:18.815335 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:18.815680 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:18.816610 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:18.816615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:18.816677 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:18.816789 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:18.817304 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:18.817316 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:18.817325 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:44:18.817519 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 21:44:18.818015 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
I20260927 21:44:18.820348 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:18.804730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:34409 (local address 127.0.202.190:39363)
0927 21:44:18.805247 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:18.805252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:18.806383 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 21:44:18.808385 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:18.808390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:18.808491 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:18.808621 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:18.810002 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:18.811323 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:18.814122 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:18.814849 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:18.814996 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:18.817089 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:18.817098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:18.817103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:18.817130 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:18.817186 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:18.817207 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:18.817315 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:18.819817 (+ 2502us) server_negotiation.cc:299] Negotiation successful
0927 21:44:18.819931 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":282,"threads_started":1}
I20260927 21:44:18.821660 19837 heartbeater.cc:347] Connected to a master server at 127.0.202.190:39363
I20260927 21:44:18.822002 19837 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:18.823009 19837 heartbeater.cc:511] Master 127.0.202.190:39363 requested a full tablet report, sending...
I20260927 21:44:18.825275 19651 ts_manager.cc:194] Registered new tserver with Master: f7868538faf34d09b440cca916349d28 (127.0.202.129:41985)
I20260927 21:44:18.826874 19651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.129:34409
W20260927 21:44:19.155926 19841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:19.156294 19841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:19.156461 19841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:19.164095 19841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:19.164258 19841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:19.164449 19841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:44:19.174041 19841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:39363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:19.175351 19841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:19.177138 19841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:19.191071 19849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:19.190990 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:19.191706 19846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:19.192147 19841 server_base.cc:1034] running on GCE node
I20260927 21:44:19.193032 19841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:19.194368 19841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:19.195626 19841 hybrid_clock.cc:648] HybridClock initialized: now 1790545459195555 us; error 61 us; skew 500 ppm
I20260927 21:44:19.196040 19841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:19.199154 19841 webserver.cc:460] Webserver started at http://127.0.202.130:43411/ using document root <none> and password file <none>
I20260927 21:44:19.200044 19841 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:19.200140 19841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:19.200562 19841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:19.203565 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "499f966abc6840ba82737e83d3bc4e67"
format_stamp: "Formatted at 2026-09-27 21:44:19 on dist-test-slave-25dq"
server_key: "3e9be818feb4707a811e1eafdf64a524"
server_key_iv: "a4e210210f03ed9aa7ae9b09513ea222"
server_key_version: "encryptionkey@0"
I20260927 21:44:19.204458 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "499f966abc6840ba82737e83d3bc4e67"
format_stamp: "Formatted at 2026-09-27 21:44:19 on dist-test-slave-25dq"
server_key: "3e9be818feb4707a811e1eafdf64a524"
server_key_iv: "a4e210210f03ed9aa7ae9b09513ea222"
server_key_version: "encryptionkey@0"
I20260927 21:44:19.209949 19841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:44:19.213737 19855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:19.215674 19841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:44:19.215900 19841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "499f966abc6840ba82737e83d3bc4e67"
format_stamp: "Formatted at 2026-09-27 21:44:19 on dist-test-slave-25dq"
server_key: "3e9be818feb4707a811e1eafdf64a524"
server_key_iv: "a4e210210f03ed9aa7ae9b09513ea222"
server_key_version: "encryptionkey@0"
I20260927 21:44:19.216144 19841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:19.237705 19841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:19.238652 19841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:19.238989 19841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:19.240067 19841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:19.241747 19841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:19.241865 19841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:19.242014 19841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:19.242110 19841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:19.278689 19841 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:37407
I20260927 21:44:19.278797 19967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:37407 every 8 connection(s)
I20260927 21:44:19.280076 19841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:44:19.283248 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19841
I20260927 21:44:19.283449 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:44:19.283885 810 external_mini_cluster.cc:1428] Setting key 14b1c232d49e5a50ab343485f54e8f0e
I20260927 21:44:19.286599 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:39363
--builtin_ntp_servers=127.0.202.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:44:19.293311 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.283046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:46863 (local address 127.0.202.190:39363)
0927 21:44:19.283277 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:19.283283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:19.284986 (+ 1703us) server_negotiation.cc:407] Connection header received
0927 21:44:19.286782 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:19.286788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:19.286875 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:19.286989 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:19.288090 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.288888 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.290455 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.290874 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.290999 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.292490 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:19.292494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:19.292498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:19.292518 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:19.292559 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:19.292570 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:19.292665 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:19.292891 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 21:44:19.292988 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:44:19.294324 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.283645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:39363 (local address 127.0.202.130:46863)
0927 21:44:19.284833 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:19.284879 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:19.286511 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:19.287229 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:19.287237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:19.287333 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:19.287925 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.287940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.289036 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.289042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.290303 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.290315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.291103 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.291111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.291340 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.292052 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:19.292055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:19.292101 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:19.292185 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:19.292665 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:19.292678 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:19.292688 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:44:19.292869 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 21:44:19.293262 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":373,"threads_started":1}
I20260927 21:44:19.295845 19968 heartbeater.cc:347] Connected to a master server at 127.0.202.190:39363
I20260927 21:44:19.296253 19968 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:19.297096 19968 heartbeater.cc:511] Master 127.0.202.190:39363 requested a full tablet report, sending...
I20260927 21:44:19.298756 19651 ts_manager.cc:194] Registered new tserver with Master: 499f966abc6840ba82737e83d3bc4e67 (127.0.202.130:37407)
I20260927 21:44:19.300043 19651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.130:46863
W20260927 21:44:19.623786 19971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:19.624094 19971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:19.624193 19971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:19.633663 19971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:19.633812 19971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:19.633935 19971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:44:19.644483 19971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:39363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:19.645871 19971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:19.647840 19971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:19.662531 19976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:19.662549 19979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:19.662653 19977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:19.664135 19971 server_base.cc:1034] running on GCE node
I20260927 21:44:19.664844 19971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:19.665863 19971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:19.667101 19971 hybrid_clock.cc:648] HybridClock initialized: now 1790545459667031 us; error 60 us; skew 500 ppm
I20260927 21:44:19.667429 19971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:19.669732 19971 webserver.cc:460] Webserver started at http://127.0.202.131:34923/ using document root <none> and password file <none>
I20260927 21:44:19.670497 19971 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:19.670626 19971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:19.670976 19971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:19.673915 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "d6a58df5090a413f988846d7c7d2bd33"
format_stamp: "Formatted at 2026-09-27 21:44:19 on dist-test-slave-25dq"
server_key: "48e8b5f63fb71d6bdfc8b7015e2acb72"
server_key_iv: "4c28800192cf67c5e8ca116c3ae4413d"
server_key_version: "encryptionkey@0"
I20260927 21:44:19.674721 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "d6a58df5090a413f988846d7c7d2bd33"
format_stamp: "Formatted at 2026-09-27 21:44:19 on dist-test-slave-25dq"
server_key: "48e8b5f63fb71d6bdfc8b7015e2acb72"
server_key_iv: "4c28800192cf67c5e8ca116c3ae4413d"
server_key_version: "encryptionkey@0"
I20260927 21:44:19.680226 19971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:44:19.684245 19985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:19.685945 19971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:19.686152 19971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "d6a58df5090a413f988846d7c7d2bd33"
format_stamp: "Formatted at 2026-09-27 21:44:19 on dist-test-slave-25dq"
server_key: "48e8b5f63fb71d6bdfc8b7015e2acb72"
server_key_iv: "4c28800192cf67c5e8ca116c3ae4413d"
server_key_version: "encryptionkey@0"
I20260927 21:44:19.686409 19971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:19.741519 19971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:19.742507 19971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:19.742864 19971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:19.743983 19971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:19.745759 19971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:19.745886 19971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:19.746017 19971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:19.746102 19971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:19.781778 19971 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:33025
I20260927 21:44:19.781939 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:33025 every 8 connection(s)
I20260927 21:44:19.783196 19971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:44:19.784621 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 19971
I20260927 21:44:19.784819 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:44:19.785244 810 external_mini_cluster.cc:1428] Setting key 62c29fdc159d3741f5e29d2b7400e158
I20260927 21:44:19.797519 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.786347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:39895 (local address 127.0.202.190:39363)
0927 21:44:19.786540 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:19.786546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:19.788132 (+ 1586us) server_negotiation.cc:407] Connection header received
0927 21:44:19.790396 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:19.790401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:19.790476 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:19.790571 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:19.791946 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.792875 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.794552 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.794940 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.795039 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.796673 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:19.796678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:19.796681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:19.796702 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:19.796737 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:19.796750 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:19.796830 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:19.797098 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 21:44:19.797191 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:44:19.798696 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.786990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:39363 (local address 127.0.202.131:39895)
0927 21:44:19.787968 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:19.788017 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:19.790002 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:19.790831 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:19.790843 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:19.791009 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:19.791763 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.791784 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.793032 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.793036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.794400 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.794412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.795155 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.795160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.795450 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.796177 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:19.796179 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:19.796223 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:19.796473 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:19.796867 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:19.796879 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:19.796889 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:44:19.797067 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 21:44:19.797485 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":321,"threads_started":1}
I20260927 21:44:19.800521 20098 heartbeater.cc:347] Connected to a master server at 127.0.202.190:39363
I20260927 21:44:19.800899 20098 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:19.801769 20098 heartbeater.cc:511] Master 127.0.202.190:39363 requested a full tablet report, sending...
I20260927 21:44:19.803452 19650 ts_manager.cc:194] Registered new tserver with Master: d6a58df5090a413f988846d7c7d2bd33 (127.0.202.131:33025)
I20260927 21:44:19.804524 19650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.131:39895
I20260927 21:44:19.814213 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:44:19.826197 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.819184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.202.190:39363)
0927 21:44:19.819397 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:19.819420 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:19.819777 (+ 357us) server_negotiation.cc:407] Connection header received
0927 21:44:19.819966 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:19.819971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:19.820039 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:19.820145 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:19.820700 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.821651 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.823383 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.823918 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.824056 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.825478 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:19.825484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:19.825487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:19.825508 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:19.825553 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:19.825566 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:19.825691 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:19.825905 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:44:19.825953 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:44:19.830114 19837 heartbeater.cc:503] Master 127.0.202.190:39363 was elected leader, sending a full tablet report...
I20260927 21:44:19.831066 19650 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:44:19.838141 19650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:44:19.875185 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.851175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:33025 (local address 127.0.0.1:47996)
0927 21:44:19.862648 (+ 11473us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:19.862706 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:19.862977 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:19.867244 (+ 4267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:19.867249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:19.867273 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:19.868140 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.868153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.869826 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.869830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.872382 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.872407 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.873979 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.873985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.874187 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.874252 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 21:44:19.874508 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 21:44:19.874751 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11091,"mutex_wait_us":354,"thread_start_us":13507,"threads_started":1}
I20260927 21:44:19.875355 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.851632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.0.202.131:33025)
0927 21:44:19.866644 (+ 15012us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:19.866650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:19.866676 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:44:19.866748 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:19.866753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:19.867003 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:19.867468 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:19.868398 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.869685 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.872662 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.873781 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.873933 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.874245 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:19.874744 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 21:44:19.874967 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14827,"thread_start_us":14666,"threads_started":1}
I20260927 21:44:19.876024 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.852219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.202.130:37407)
0927 21:44:19.865791 (+ 13572us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:19.865802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:19.865841 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:44:19.865959 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:19.865969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:19.866307 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:19.866554 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:19.867607 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.869316 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.872667 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.874238 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.874384 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.874621 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:19.875291 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 21:44:19.875765 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13392,"thread_start_us":13257,"threads_started":1}
I20260927 21:44:19.876245 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.854043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.202.129:41985)
0927 21:44:19.865277 (+ 11234us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:19.865291 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:19.866288 (+ 997us) server_negotiation.cc:407] Connection header received
0927 21:44:19.866597 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:19.866607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:19.866859 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:19.867027 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:19.868059 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.869594 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.872334 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.873832 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.873942 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.874236 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:19.875283 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0927 21:44:19.875843 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10961,"thread_start_us":10776,"threads_started":1}
I20260927 21:44:19.876669 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.853803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35624)
0927 21:44:19.866138 (+ 12335us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:19.866167 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:19.866400 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:19.867371 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:19.867374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:19.867402 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:19.867854 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.867870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.869720 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.869727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.872053 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.872071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.874551 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.874560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.874747 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.874842 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 21:44:19.874928 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:44:19.875070 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12196,"mutex_wait_us":169,"thread_start_us":8252,"threads_started":1}
I20260927 21:44:19.877174 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:19.851570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37407 (local address 127.0.0.1:40198)
0927 21:44:19.864970 (+ 13400us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:19.864991 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:19.865141 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:19.866850 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:19.866858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:19.866890 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:19.867366 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.867377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.869470 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.869476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.872431 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:19.872447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:19.874572 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:19.874579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:19.874846 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:19.874880 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:44:19.874968 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:44:19.875095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13263,"mutex_wait_us":186,"thread_start_us":14273,"threads_started":1}
I20260927 21:44:19.879518 20033 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab08810d00042ef977f5f7df00f65a4 (DEFAULT_TABLE table=test-table [id=3843c45103db4a159cf4318dc52804fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:19.881054 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab08810d00042ef977f5f7df00f65a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:19.883172 19772 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab08810d00042ef977f5f7df00f65a4 (DEFAULT_TABLE table=test-table [id=3843c45103db4a159cf4318dc52804fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:19.883574 19903 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab08810d00042ef977f5f7df00f65a4 (DEFAULT_TABLE table=test-table [id=3843c45103db4a159cf4318dc52804fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:19.884737 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab08810d00042ef977f5f7df00f65a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:19.885115 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab08810d00042ef977f5f7df00f65a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:19.898530 20117 tablet_bootstrap.cc:492] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Bootstrap starting.
I20260927 21:44:19.901077 20118 tablet_bootstrap.cc:492] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Bootstrap starting.
I20260927 21:44:19.902770 20116 tablet_bootstrap.cc:492] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Bootstrap starting.
I20260927 21:44:19.903381 20117 tablet_bootstrap.cc:654] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:19.905251 20117 log.cc:824] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:19.905610 20118 tablet_bootstrap.cc:654] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:19.906816 20116 tablet_bootstrap.cc:654] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:19.906996 20118 log.cc:824] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:19.907992 20116 log.cc:824] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:19.910372 20118 tablet_bootstrap.cc:492] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: No bootstrap required, opened a new log
I20260927 21:44:19.910652 20118 ts_tablet_manager.cc:1397] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 21:44:19.911165 20116 tablet_bootstrap.cc:492] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: No bootstrap required, opened a new log
I20260927 21:44:19.911182 20117 tablet_bootstrap.cc:492] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: No bootstrap required, opened a new log
I20260927 21:44:19.911454 20116 ts_tablet_manager.cc:1397] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 21:44:19.911453 20117 ts_tablet_manager.cc:1397] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260927 21:44:19.918562 20118 raft_consensus.cc:348] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:19.918975 20118 raft_consensus.cc:374] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:19.919065 20118 raft_consensus.cc:729] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7868538faf34d09b440cca916349d28, State: Initialized, Role: FOLLOWER
I20260927 21:44:19.919799 20118 consensus_queue.cc:260] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:19.921813 20118 ts_tablet_manager.cc:1428] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260927 21:44:19.922542 20117 raft_consensus.cc:348] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:19.922542 20116 raft_consensus.cc:348] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:19.923020 20117 raft_consensus.cc:374] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:19.923022 20116 raft_consensus.cc:374] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:19.923130 20116 raft_consensus.cc:729] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a58df5090a413f988846d7c7d2bd33, State: Initialized, Role: FOLLOWER
I20260927 21:44:19.923130 20117 raft_consensus.cc:729] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499f966abc6840ba82737e83d3bc4e67, State: Initialized, Role: FOLLOWER
I20260927 21:44:19.923765 20116 consensus_queue.cc:260] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:19.923764 20117 consensus_queue.cc:260] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:19.925571 20098 heartbeater.cc:503] Master 127.0.202.190:39363 was elected leader, sending a full tablet report...
I20260927 21:44:19.925678 20117 ts_tablet_manager.cc:1428] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 21:44:19.925938 20116 ts_tablet_manager.cc:1428] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260927 21:44:19.926126 19968 heartbeater.cc:503] Master 127.0.202.190:39363 was elected leader, sending a full tablet report...
W20260927 21:44:20.032253 19969 tablet.cc:2369] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:44:20.035328 20099 tablet.cc:2369] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:44:20.054819 19838 tablet.cc:2369] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:20.055596 20123 raft_consensus.cc:482] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:44:20.055929 20123 raft_consensus.cc:504] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:20.057868 20123 leader_election.cc:288] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499f966abc6840ba82737e83d3bc4e67 (127.0.202.130:37407), f7868538faf34d09b440cca916349d28 (127.0.202.129:41985)
I20260927 21:44:20.064713 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.058560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:60057 (local address 127.0.202.130:37407)
0927 21:44:20.058789 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:20.058797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:20.058820 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:44:20.058913 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:20.058920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:20.059021 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:20.059126 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:20.059680 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.060847 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.062697 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.063731 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.063855 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.063914 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:20.064337 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 21:44:20.064507 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:44:20.064849 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.058090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:37407 (local address 127.0.202.131:60057)
0927 21:44:20.058317 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.058342 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:20.058486 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:20.059231 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:20.059236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:20.059275 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:20.059490 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:20.059507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.061019 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.061023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:20.062523 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:20.062536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.063895 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.063901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:20.064122 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.064162 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:44:20.064275 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 21:44:20.064467 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 21:44:20.065763 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.058207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.202.131:49709)
0927 21:44:20.059221 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.059244 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:20.059399 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:20.059786 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:20.059790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:20.059815 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:20.060178 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:20.060190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.061534 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.061544 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:20.063187 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:20.063201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.065102 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.065111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:20.065297 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.065328 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:44:20.065442 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:44:20.065557 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":613,"threads_started":1}
I20260927 21:44:20.065954 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.058452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:49709 (local address 127.0.202.129:41985)
0927 21:44:20.058711 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:20.058720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:20.059332 (+ 612us) server_negotiation.cc:407] Connection header received
0927 21:44:20.059552 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:20.059558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:20.059655 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:20.059754 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:20.060437 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.061374 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.063480 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.064724 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.064898 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.064986 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:20.065528 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 21:44:20.065699 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 21:44:20.066087 19923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab08810d00042ef977f5f7df00f65a4" candidate_uuid: "d6a58df5090a413f988846d7c7d2bd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499f966abc6840ba82737e83d3bc4e67" is_pre_election: true
I20260927 21:44:20.066674 19923 raft_consensus.cc:2454] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a58df5090a413f988846d7c7d2bd33 in term 0.
I20260927 21:44:20.066885 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab08810d00042ef977f5f7df00f65a4" candidate_uuid: "d6a58df5090a413f988846d7c7d2bd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7868538faf34d09b440cca916349d28" is_pre_election: true
I20260927 21:44:20.067402 19792 raft_consensus.cc:2454] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a58df5090a413f988846d7c7d2bd33 in term 0.
I20260927 21:44:20.067415 19988 leader_election.cc:302] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499f966abc6840ba82737e83d3bc4e67, d6a58df5090a413f988846d7c7d2bd33; no voters:
I20260927 21:44:20.067821 20123 raft_consensus.cc:2781] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:44:20.067997 20123 raft_consensus.cc:482] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:44:20.068135 20123 raft_consensus.cc:3037] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:20.070580 20123 raft_consensus.cc:504] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:20.071257 20123 leader_election.cc:288] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [CANDIDATE]: Term 1 election: Requested vote from peers 499f966abc6840ba82737e83d3bc4e67 (127.0.202.130:37407), f7868538faf34d09b440cca916349d28 (127.0.202.129:41985)
I20260927 21:44:20.071817 19923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab08810d00042ef977f5f7df00f65a4" candidate_uuid: "d6a58df5090a413f988846d7c7d2bd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499f966abc6840ba82737e83d3bc4e67"
I20260927 21:44:20.071950 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab08810d00042ef977f5f7df00f65a4" candidate_uuid: "d6a58df5090a413f988846d7c7d2bd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7868538faf34d09b440cca916349d28"
I20260927 21:44:20.072041 19923 raft_consensus.cc:3037] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:20.072196 19792 raft_consensus.cc:3037] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:20.074513 19923 raft_consensus.cc:2454] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a58df5090a413f988846d7c7d2bd33 in term 1.
I20260927 21:44:20.074877 19792 raft_consensus.cc:2454] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a58df5090a413f988846d7c7d2bd33 in term 1.
I20260927 21:44:20.074956 19988 leader_election.cc:302] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499f966abc6840ba82737e83d3bc4e67, d6a58df5090a413f988846d7c7d2bd33; no voters:
I20260927 21:44:20.075445 20123 raft_consensus.cc:2781] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:20.075924 20123 raft_consensus.cc:686] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 1 LEADER]: Becoming Leader. State: Replica: d6a58df5090a413f988846d7c7d2bd33, State: Running, Role: LEADER
I20260927 21:44:20.076538 20123 consensus_queue.cc:237] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } }
I20260927 21:44:20.081759 19650 catalog_manager.cc:5495] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6a58df5090a413f988846d7c7d2bd33 (127.0.202.131). New cstate: current_term: 1 leader_uuid: "d6a58df5090a413f988846d7c7d2bd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a58df5090a413f988846d7c7d2bd33" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 33025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 } health_report { overall_health: UNKNOWN } } }
I20260927 21:44:20.119619 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.113905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.0.202.131:33025)
0927 21:44:20.114122 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:20.114128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:20.114147 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:44:20.114306 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:20.114310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:20.114397 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:20.114536 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:44:20.114883 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.115615 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.117112 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.117691 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.117797 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.118388 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:44:20.118972 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:44:20.119106 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:20.119296 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 21:44:20.119335 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:44:20.128748 19792 raft_consensus.cc:1261] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 1 FOLLOWER]: Refusing update from remote peer d6a58df5090a413f988846d7c7d2bd33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:20.128790 19923 raft_consensus.cc:1261] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 1 FOLLOWER]: Refusing update from remote peer d6a58df5090a413f988846d7c7d2bd33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:20.129730 20126 consensus_queue.cc:1035] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499f966abc6840ba82737e83d3bc4e67" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 37407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:20.130165 20123 consensus_queue.cc:1035] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7868538faf34d09b440cca916349d28" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:20.156785 19651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:20.157055 19651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:20.160794 19651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4d102a5b2b848fe968ff40ec5487832: Sending DeleteTablet for 3 replicas of tablet 1ab08810d00042ef977f5f7df00f65a4
I20260927 21:44:20.162006 20033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab08810d00042ef977f5f7df00f65a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:20 UTC) from {username='slave'} at 127.0.0.1:47996
I20260927 21:44:20.162096 19903 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab08810d00042ef977f5f7df00f65a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:20 UTC) from {username='slave'} at 127.0.0.1:40198
I20260927 21:44:20.162096 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab08810d00042ef977f5f7df00f65a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:20 UTC) from {username='slave'} at 127.0.0.1:35624
I20260927 21:44:20.165341 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19710
I20260927 21:44:20.175351 20139 tablet_replica.cc:330] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: stopping tablet replica
I20260927 21:44:20.176002 20139 raft_consensus.cc:2229] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:20.176677 20139 raft_consensus.cc:2258] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:20.177657 20140 tablet_replica.cc:330] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: stopping tablet replica
I20260927 21:44:20.178221 20140 raft_consensus.cc:2229] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:20.178478 20138 tablet_replica.cc:330] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: stopping tablet replica
I20260927 21:44:20.178810 20140 raft_consensus.cc:2258] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:20.181770 20139 ts_tablet_manager.cc:1905] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:20.189993 20140 ts_tablet_manager.cc:1905] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:20.190608 20139 ts_tablet_manager.cc:1918] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:20.190899 20139 log.cc:1196] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/1ab08810d00042ef977f5f7df00f65a4
I20260927 21:44:20.191627 20139 ts_tablet_manager.cc:1939] T 1ab08810d00042ef977f5f7df00f65a4 P 499f966abc6840ba82737e83d3bc4e67: Deleting consensus metadata
I20260927 21:44:20.193954 19637 catalog_manager.cc:4841] TS 499f966abc6840ba82737e83d3bc4e67 (127.0.202.130:37407): tablet 1ab08810d00042ef977f5f7df00f65a4 (table test-table [id=3843c45103db4a159cf4318dc52804fe]) successfully deleted
I20260927 21:44:20.196902 20138 raft_consensus.cc:2229] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:44:20.197947 20138 raft_consensus.cc:2258] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:20.198642 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.167255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.0.202.129:41985)
0927 21:44:20.167477 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:20.167483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:20.179875 (+ 12392us) server_negotiation.cc:407] Connection header received
0927 21:44:20.180192 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:20.180198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:20.180429 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:20.180525 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:20.181516 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.182762 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.185129 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.185828 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.185938 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.196569 (+ 10631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:20.196582 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:20.196602 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:20.197501 (+ 899us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:20.197578 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:20.197610 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:20.197747 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:20.198108 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 21:44:20.198252 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:44:20.200047 19832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:20.201009 20138 ts_tablet_manager.cc:1905] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:20.209455 20138 ts_tablet_manager.cc:1918] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:20.209787 20138 log.cc:1196] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/1ab08810d00042ef977f5f7df00f65a4
I20260927 21:44:20.210475 20138 ts_tablet_manager.cc:1939] T 1ab08810d00042ef977f5f7df00f65a4 P d6a58df5090a413f988846d7c7d2bd33: Deleting consensus metadata
I20260927 21:44:20.212788 19638 catalog_manager.cc:4841] TS d6a58df5090a413f988846d7c7d2bd33 (127.0.202.131:33025): tablet 1ab08810d00042ef977f5f7df00f65a4 (table test-table [id=3843c45103db4a159cf4318dc52804fe]) successfully deleted
I20260927 21:44:20.304742 20140 ts_tablet_manager.cc:1918] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:20.304997 20140 log.cc:1196] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/1ab08810d00042ef977f5f7df00f65a4
I20260927 21:44:20.305497 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19710
I20260927 21:44:20.305683 20140 ts_tablet_manager.cc:1939] T 1ab08810d00042ef977f5f7df00f65a4 P f7868538faf34d09b440cca916349d28: Deleting consensus metadata
W20260927 21:44:20.324550 19635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.202.129:41985 (error 108)
I20260927 21:44:20.324702 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19841
I20260927 21:44:20.326594 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.326092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35638)
0927 21:44:20.326258 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.326395 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:44:20.327036 19635 catalog_manager.cc:4584] TS f7868538faf34d09b440cca916349d28 (127.0.202.129:41985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ab08810d00042ef977f5f7df00f65a4: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
I20260927 21:44:20.334173 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.326207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.202.130:37407)
0927 21:44:20.326385 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:20.326392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:20.326413 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:44:20.326479 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:20.326484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:20.326629 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:20.326740 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:20.327492 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.328452 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.330622 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.331118 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.331236 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.332627 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:20.332637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:20.332654 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:20.333331 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:20.333390 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:20.333412 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:20.333509 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:20.333681 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 21:44:20.333806 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:44:20.334906 19963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:20.380784 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.380178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35650)
0927 21:44:20.380472 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.380598 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 21:44:20.382090 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.381646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35656)
0927 21:44:20.381828 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.381919 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 21:44:20.426672 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19841
I20260927 21:44:20.443183 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.442550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35670)
0927 21:44:20.442820 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.442950 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:44:20.444510 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.443978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35672)
0927 21:44:20.444207 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.444332 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 21:44:20.444624 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19971
I20260927 21:44:20.452906 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.445451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.0.202.131:33025)
0927 21:44:20.445607 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:20.445612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:20.445629 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:44:20.445734 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:20.445738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:20.445819 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:20.445901 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:20.446430 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.447230 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.449325 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:20.449746 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:20.449876 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:20.451248 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:20.451260 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:20.451279 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:20.451938 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:20.452003 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:20.452027 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:20.452121 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:20.452413 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:44:20.452501 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:44:20.453624 20093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:20.525362 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.524718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35674)
0927 21:44:20.524974 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.525138 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 21:44:20.526917 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.526338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35688)
0927 21:44:20.526658 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.526776 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 21:44:20.581204 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19971
I20260927 21:44:20.602568 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 19620
I20260927 21:44:20.603758 19681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:20.720432 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:20.719112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:41985 (local address 127.0.0.1:35700)
0927 21:44:20.719443 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:20.719530 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 21:44:20.721328 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 19620
2026-09-27T21:44:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:20 dist-test-slave-25dq krb5kdc[20149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:20 dist-test-slave-25dq krb5kdc[20149](info): set up 2 sockets
Sep 27 21:44:20 dist-test-slave-25dq krb5kdc[20149](info): commencing operation
krb5kdc: starting...
W20260927 21:44:21.856688 810 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:22 dist-test-slave-25dq krb5kdc[20149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:44:22.125795 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35841
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:35841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:22.481513 20162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:22.481827 20162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:22.481936 20162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:22.489593 20162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:22.489733 20162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:22.489780 20162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:22.489831 20162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:22.489903 20162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:22.500149 20162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:35841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:22.501482 20162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:22.502861 20162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:22.515491 20167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:22.515551 20170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:22.515523 20168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:22.516129 20162 server_base.cc:1034] running on GCE node
I20260927 21:44:22.517145 20162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:22.518733 20162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:22.520001 20162 hybrid_clock.cc:648] HybridClock initialized: now 1790545462519938 us; error 55 us; skew 500 ppm
I20260927 21:44:22.520443 20162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:22 dist-test-slave-25dq krb5kdc[20149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:22.523797 20162 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:44:22.525333 20162 webserver.cc:460] Webserver started at http://127.0.202.190:40511/ using document root <none> and password file <none>
I20260927 21:44:22.526011 20162 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:22.526114 20162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:22.526398 20162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:22.529476 20162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "c754c08acc7048f4bab8af8d990be399"
format_stamp: "Formatted at 2026-09-27 21:44:22 on dist-test-slave-25dq"
server_key: "c89a109b4102b837ef2b052fb3f6952b"
server_key_iv: "0e85242c696556164f6eb8cc6a21247c"
server_key_version: "encryptionkey@0"
I20260927 21:44:22.530203 20162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "c754c08acc7048f4bab8af8d990be399"
format_stamp: "Formatted at 2026-09-27 21:44:22 on dist-test-slave-25dq"
server_key: "c89a109b4102b837ef2b052fb3f6952b"
server_key_iv: "0e85242c696556164f6eb8cc6a21247c"
server_key_version: "encryptionkey@0"
I20260927 21:44:22.536659 20162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 21:44:22.540228 20177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:22.541816 20162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:22.542021 20162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "c754c08acc7048f4bab8af8d990be399"
format_stamp: "Formatted at 2026-09-27 21:44:22 on dist-test-slave-25dq"
server_key: "c89a109b4102b837ef2b052fb3f6952b"
server_key_iv: "0e85242c696556164f6eb8cc6a21247c"
server_key_version: "encryptionkey@0"
I20260927 21:44:22.542244 20162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:22.563699 20162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:22.568378 20162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:22.568648 20162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:22.588673 20162 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:35841
I20260927 21:44:22.588691 20228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:35841 every 8 connection(s)
I20260927 21:44:22.590350 20162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:22.593379 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 20162
I20260927 21:44:22.593575 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:22.594003 810 external_mini_cluster.cc:1428] Setting key e2b03ab16b28921dc5012f0599dcbf01
I20260927 21:44:22.595129 20229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:22.607937 20229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399: Bootstrap starting.
Sep 27 21:44:22 dist-test-slave-25dq krb5kdc[20149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:22.611825 20229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:22.613009 20229 log.cc:824] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:22.616322 20229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399: No bootstrap required, opened a new log
I20260927 21:44:22.618568 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:22.595869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.0.202.190:35841)
0927 21:44:22.596793 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:22.596811 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:22.597144 (+ 333us) server_negotiation.cc:407] Connection header received
0927 21:44:22.598865 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:22.598888 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:22.599299 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:22.599721 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:22.601353 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:22.602689 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:22.603952 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:22.604747 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:22.604991 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:22.610015 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:22.610027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:22.610047 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:22.610102 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:22.613981 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:22.614902 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:22.614909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:22.614916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:22.615093 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:22.615562 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:22.615566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:22.615568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:22.616143 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:22.616603 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:22.616950 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 21:44:22.617394 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":402,"threads_started":1}
I20260927 21:44:22.625833 20229 raft_consensus.cc:348] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c754c08acc7048f4bab8af8d990be399" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35841 } }
I20260927 21:44:22.626236 20229 raft_consensus.cc:374] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:22.626312 20229 raft_consensus.cc:729] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c754c08acc7048f4bab8af8d990be399, State: Initialized, Role: FOLLOWER
I20260927 21:44:22.627051 20229 consensus_queue.cc:260] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c754c08acc7048f4bab8af8d990be399" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35841 } }
I20260927 21:44:22.627403 20229 raft_consensus.cc:388] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:22.627571 20229 raft_consensus.cc:482] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:22.627768 20229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:22.631373 20229 raft_consensus.cc:504] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c754c08acc7048f4bab8af8d990be399" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35841 } }
I20260927 21:44:22.632156 20229 leader_election.cc:302] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c754c08acc7048f4bab8af8d990be399; no voters:
I20260927 21:44:22.633024 20229 leader_election.cc:288] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:22.633225 20234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:22.635423 20229 sys_catalog.cc:564] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:22.637329 20234 raft_consensus.cc:686] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [term 1 LEADER]: Becoming Leader. State: Replica: c754c08acc7048f4bab8af8d990be399, State: Running, Role: LEADER
I20260927 21:44:22.638007 20234 consensus_queue.cc:237] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c754c08acc7048f4bab8af8d990be399" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35841 } }
I20260927 21:44:22.641121 20235 sys_catalog.cc:455] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c754c08acc7048f4bab8af8d990be399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c754c08acc7048f4bab8af8d990be399" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35841 } } }
I20260927 21:44:22.641525 20235 sys_catalog.cc:458] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:22.641250 20236 sys_catalog.cc:455] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c754c08acc7048f4bab8af8d990be399. Latest consensus state: current_term: 1 leader_uuid: "c754c08acc7048f4bab8af8d990be399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c754c08acc7048f4bab8af8d990be399" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35841 } } }
I20260927 21:44:22.641657 20236 sys_catalog.cc:458] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:22.648082 20241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:22.653702 20241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:22.663369 20241 catalog_manager.cc:1347] Generated new cluster ID: 1742c0e739bb476a8864e11ac74d6ca3
I20260927 21:44:22.663518 20241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:22.680588 20241 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:22.681957 20241 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:22.687991 20241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399: Generated new TSK 0
I20260927 21:44:22.688948 20241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
Entry for principal kudu/127.0.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab.
I20260927 21:44:22.759819 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35841
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:23.112262 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:23.112637 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:23.112736 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:23.120203 20255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:23.120385 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:23.120523 20255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:44:23.130376 20255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.129.keytab
--principal=kudu/127.0.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:23.131623 20255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:23.133070 20255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:23.146270 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:23.147533 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:23.148005 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:23.148936 20255 server_base.cc:1034] running on GCE node
I20260927 21:44:23.149663 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:23.150666 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:23.151868 20255 hybrid_clock.cc:648] HybridClock initialized: now 1790545463151799 us; error 63 us; skew 500 ppm
I20260927 21:44:23.152196 20255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:23 dist-test-slave-25dq krb5kdc[20149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:23.161919 20255 init.cc:377] Logged in from keytab as kudu/127.0.202.129@KRBTEST.COM (short username kudu)
I20260927 21:44:23.163630 20255 webserver.cc:460] Webserver started at http://127.0.202.129:36161/ using document root <none> and password file <none>
I20260927 21:44:23.164460 20255 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:23.164614 20255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:23.164937 20255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:23.167941 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49"
format_stamp: "Formatted at 2026-09-27 21:44:23 on dist-test-slave-25dq"
server_key: "d1b3317a545f8f61082bdfdd9ac770b7"
server_key_iv: "f9f7bed4551f0dd09614326081d9cec2"
server_key_version: "encryptionkey@0"
I20260927 21:44:23.168797 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49"
format_stamp: "Formatted at 2026-09-27 21:44:23 on dist-test-slave-25dq"
server_key: "d1b3317a545f8f61082bdfdd9ac770b7"
server_key_iv: "f9f7bed4551f0dd09614326081d9cec2"
server_key_version: "encryptionkey@0"
I20260927 21:44:23.174574 20255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:44:23.178617 20270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:23.180094 20255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:44:23.180346 20255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49"
format_stamp: "Formatted at 2026-09-27 21:44:23 on dist-test-slave-25dq"
server_key: "d1b3317a545f8f61082bdfdd9ac770b7"
server_key_iv: "f9f7bed4551f0dd09614326081d9cec2"
server_key_version: "encryptionkey@0"
I20260927 21:44:23.180572 20255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:23.212179 20255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:23.216876 20255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:23.217293 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:23.218446 20255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:23.219872 20255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:23.219982 20255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:23.220139 20255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:23.220221 20255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:23.255383 20255 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:34427
I20260927 21:44:23.255482 20382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:34427 every 8 connection(s)
I20260927 21:44:23.256832 20255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:44:23.257223 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 20255
I20260927 21:44:23.257445 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:44:23.258122 810 external_mini_cluster.cc:1428] Setting key fb991b507e75a54b2201f5f7b0ed5a9d
Sep 27 21:44:23 dist-test-slave-25dq krb5kdc[20149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.129@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:23.283156 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:23.259618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:53137 (local address 127.0.202.190:35841)
0927 21:44:23.260104 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:23.260110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:23.261492 (+ 1382us) server_negotiation.cc:407] Connection header received
0927 21:44:23.264880 (+ 3388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:23.264884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:23.264956 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:23.265054 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:23.267419 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.268140 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.271088 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.271861 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.272008 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:23.276264 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:23.276270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:23.276274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:23.276386 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:23.279178 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:23.280285 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:23.280347 (+ 62us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:23.280354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:23.280511 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:23.281057 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:23.281061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:23.281063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:23.281434 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:23.281551 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:23.282567 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0927 21:44:23.282825 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260927 21:44:23.284904 20386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:23.260392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35841 (local address 127.0.202.129:53137)
0927 21:44:23.261292 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:23.261356 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:23.264524 (+ 3168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:23.265372 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:23.265387 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:23.266274 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:23.267194 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:23.267226 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.268331 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.268340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:23.270860 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:23.270880 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.272103 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.272108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:23.272447 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:23.273224 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:23.273248 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:23.275931 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:23.279347 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:23.279359 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:23.279381 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:23.280113 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:23.280635 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:23.280643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:23.280649 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:23.280927 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:23.281572 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:23.281584 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:23.282101 (+ 517us) client_negotiation.cc:769] Sending connection context
0927 21:44:23.282468 (+ 367us) client_negotiation.cc:240] Negotiation successful
0927 21:44:23.283149 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":331,"threads_started":1}
I20260927 21:44:23.286921 20383 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35841
I20260927 21:44:23.287366 20383 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:23.288235 20383 heartbeater.cc:511] Master 127.0.202.190:35841 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.202.130@KRBTEST.COM; defaulting to no policy
I20260927 21:44:23.290895 20194 ts_manager.cc:194] Registered new tserver with Master: c97c7a5e6b2d40c1a657d9cc5d9c4a49 (127.0.202.129:34427)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.130@KRBTEST.COM" created.
I20260927 21:44:23.293380 20194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.129@KRBTEST.COM'} at 127.0.202.129:53137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
Entry for principal kudu/127.0.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab.
I20260927 21:44:23.324857 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35841
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:23.664019 20389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:23.664410 20389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:23.664592 20389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:23.672518 20389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:23.672640 20389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:23.672756 20389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:44:23.682777 20389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.130.keytab
--principal=kudu/127.0.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:23.684322 20389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:23.685776 20389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:23.699283 20395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:23.699397 20394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:23.699304 20397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:23.705760 20389 server_base.cc:1034] running on GCE node
I20260927 21:44:23.706458 20389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:23.707512 20389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:23.708743 20389 hybrid_clock.cc:648] HybridClock initialized: now 1790545463708704 us; error 57 us; skew 500 ppm
I20260927 21:44:23.709081 20389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:23 dist-test-slave-25dq krb5kdc[20149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:23.712466 20389 init.cc:377] Logged in from keytab as kudu/127.0.202.130@KRBTEST.COM (short username kudu)
I20260927 21:44:23.714143 20389 webserver.cc:460] Webserver started at http://127.0.202.130:38665/ using document root <none> and password file <none>
I20260927 21:44:23.714897 20389 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:23.715044 20389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:23.715366 20389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:23.718405 20389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "47131b4d98134b09ba5dcacee796ae07"
format_stamp: "Formatted at 2026-09-27 21:44:23 on dist-test-slave-25dq"
server_key: "c445002d3b72183a1b120cb24c63962e"
server_key_iv: "e667d8b9e44de67747960611805a1e83"
server_key_version: "encryptionkey@0"
I20260927 21:44:23.719208 20389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "47131b4d98134b09ba5dcacee796ae07"
format_stamp: "Formatted at 2026-09-27 21:44:23 on dist-test-slave-25dq"
server_key: "c445002d3b72183a1b120cb24c63962e"
server_key_iv: "e667d8b9e44de67747960611805a1e83"
server_key_version: "encryptionkey@0"
I20260927 21:44:23.724720 20389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:44:23.728466 20404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:23.730154 20389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:23.730355 20389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "47131b4d98134b09ba5dcacee796ae07"
format_stamp: "Formatted at 2026-09-27 21:44:23 on dist-test-slave-25dq"
server_key: "c445002d3b72183a1b120cb24c63962e"
server_key_iv: "e667d8b9e44de67747960611805a1e83"
server_key_version: "encryptionkey@0"
I20260927 21:44:23.730578 20389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:23.753587 20389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:23.758313 20389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:23.758658 20389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:23.759701 20389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:23.761343 20389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:23.761452 20389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:23.761596 20389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:23.761668 20389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:23.796543 20389 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:33053
I20260927 21:44:23.796604 20516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:33053 every 8 connection(s)
I20260927 21:44:23.797873 20389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:44:23.801337 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 20389
I20260927 21:44:23.801560 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:44:23.802120 810 external_mini_cluster.cc:1428] Setting key ee6f2a0711583210313826986649bc04
Sep 27 21:44:23 dist-test-slave-25dq krb5kdc[20149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.130@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:23.819763 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:23.800565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:52283 (local address 127.0.202.190:35841)
0927 21:44:23.801086 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:23.801090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:23.802149 (+ 1059us) server_negotiation.cc:407] Connection header received
0927 21:44:23.804277 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:23.804282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:23.804391 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:23.804513 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:23.806027 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.806731 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.808336 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.808769 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.808904 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:23.812657 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:23.812666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:23.812674 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:23.812721 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:23.816283 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:23.817070 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:23.817079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:23.817085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:23.817254 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:23.817668 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:23.817675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:23.817681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:23.818070 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:23.818211 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:23.818962 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 21:44:23.819242 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":293,"threads_started":1}
I20260927 21:44:23.820632 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:23.801151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35841 (local address 127.0.202.130:52283)
0927 21:44:23.801985 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:23.802029 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:23.804021 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:23.804749 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:23.804758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:23.805288 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:23.805848 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:23.805869 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.806896 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.806901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:23.808155 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:23.808167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:23.808994 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:23.809000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:23.809233 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:23.810042 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:23.810065 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:23.812415 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:23.816462 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:23.816469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:23.816483 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:23.816924 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:23.817372 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:23.817375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:23.817378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:23.817558 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:23.818232 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:23.818238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:23.818583 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 21:44:23.818834 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 21:44:23.819264 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":346,"threads_started":1}
I20260927 21:44:23.822582 20517 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35841
I20260927 21:44:23.822914 20517 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:23.823822 20517 heartbeater.cc:511] Master 127.0.202.190:35841 requested a full tablet report, sending...
I20260927 21:44:23.825747 20194 ts_manager.cc:194] Registered new tserver with Master: 47131b4d98134b09ba5dcacee796ae07 (127.0.202.130:33053)
I20260927 21:44:23.826993 20194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.130@KRBTEST.COM'} at 127.0.202.130:52283
WARNING: no policy specified for kudu/127.0.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
Entry for principal kudu/127.0.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab.
I20260927 21:44:23.864686 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35841
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:24.201877 20523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:24.202178 20523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:24.202333 20523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:24.210215 20523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:24.210423 20523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:24.210685 20523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:44:24.221549 20523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/krb5kdc/kudu_127.0.202.131.keytab
--principal=kudu/127.0.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:24.222860 20523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:24.224145 20523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:24.238260 20529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:24.238324 20531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:24.238346 20528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:24.238915 20523 server_base.cc:1034] running on GCE node
I20260927 21:44:24.239639 20523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:24.240712 20523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:24.241945 20523 hybrid_clock.cc:648] HybridClock initialized: now 1790545464241874 us; error 68 us; skew 500 ppm
I20260927 21:44:24.242264 20523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:24 dist-test-slave-25dq krb5kdc[20149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:24.245385 20523 init.cc:377] Logged in from keytab as kudu/127.0.202.131@KRBTEST.COM (short username kudu)
I20260927 21:44:24.246977 20523 webserver.cc:460] Webserver started at http://127.0.202.131:35873/ using document root <none> and password file <none>
I20260927 21:44:24.247697 20523 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:24.247893 20523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:24.248256 20523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:24.251359 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "7729a85238944feebc4f366fdf0ca909"
format_stamp: "Formatted at 2026-09-27 21:44:24 on dist-test-slave-25dq"
server_key: "3be067178071012bc66bc0a5c4a8f3f7"
server_key_iv: "86c51db6502af3f70c3e53ad27dfa5ab"
server_key_version: "encryptionkey@0"
I20260927 21:44:24.252149 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "7729a85238944feebc4f366fdf0ca909"
format_stamp: "Formatted at 2026-09-27 21:44:24 on dist-test-slave-25dq"
server_key: "3be067178071012bc66bc0a5c4a8f3f7"
server_key_iv: "86c51db6502af3f70c3e53ad27dfa5ab"
server_key_version: "encryptionkey@0"
I20260927 21:44:24.257551 20523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 21:44:24.261132 20538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:24.262588 20523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:24.262766 20523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "7729a85238944feebc4f366fdf0ca909"
format_stamp: "Formatted at 2026-09-27 21:44:24 on dist-test-slave-25dq"
server_key: "3be067178071012bc66bc0a5c4a8f3f7"
server_key_iv: "86c51db6502af3f70c3e53ad27dfa5ab"
server_key_version: "encryptionkey@0"
I20260927 21:44:24.262995 20523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:24.296808 20523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:24.297840 20383 heartbeater.cc:503] Master 127.0.202.190:35841 was elected leader, sending a full tablet report...
I20260927 21:44:24.301976 20523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:24.302270 20523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:24.303372 20523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:24.304817 20523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:24.304924 20523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:24.305083 20523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:24.305154 20523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:24.338970 20523 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:41877
I20260927 21:44:24.339015 20650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:41877 every 8 connection(s)
I20260927 21:44:24.340411 20523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:44:24.341653 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 20523
I20260927 21:44:24.341837 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:44:24.342371 810 external_mini_cluster.cc:1428] Setting key 11ca4d3daa5b2b01ec41ea8fee82d9dd
Sep 27 21:44:24 dist-test-slave-25dq krb5kdc[20149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.131@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:24.363317 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.343577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:50717 (local address 127.0.202.190:35841)
0927 21:44:24.348666 (+ 5089us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.348671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.348688 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:44:24.349271 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.349275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.349360 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.349484 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:24.351145 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.351934 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.353555 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.354021 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.354126 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.357656 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:24.357660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:24.357664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:24.357695 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:24.360361 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.361059 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.361064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.361068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.361187 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.361598 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.361603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.361606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.361877 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:24.362031 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.362717 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.362936 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4991,"thread_start_us":4858,"threads_started":1}
I20260927 21:44:24.364274 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.343997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35841 (local address 127.0.202.131:50717)
0927 21:44:24.346015 (+ 2018us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.346080 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:24.349010 (+ 2930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:24.349835 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:24.349844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:24.350410 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:24.351002 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.351018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.352079 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.352083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.353422 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.353434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.354230 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.354236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.354439 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.355075 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:44:24.355098 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:24.357447 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:24.360489 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:24.360498 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:24.360516 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:24.360934 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:24.361300 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:44:24.361304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:44:24.361308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:44:24.361483 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:44:24.362003 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:24.362010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:24.362364 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 21:44:24.362621 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 21:44:24.363129 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":403,"threads_started":1}
I20260927 21:44:24.365954 20651 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35841
I20260927 21:44:24.366299 20651 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:24.367038 20651 heartbeater.cc:511] Master 127.0.202.190:35841 requested a full tablet report, sending...
I20260927 21:44:24.368551 20194 ts_manager.cc:194] Registered new tserver with Master: 7729a85238944feebc4f366fdf0ca909 (127.0.202.131:41877)
I20260927 21:44:24.369514 20194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.202.131@KRBTEST.COM'} at 127.0.202.131:50717
I20260927 21:44:24.370636 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:44:24.389158 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.375701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.0.202.190:35841)
0927 21:44:24.375891 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.375895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.376192 (+ 297us) server_negotiation.cc:407] Connection header received
0927 21:44:24.376383 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.376388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.376473 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.376601 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:24.378032 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.378809 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.380215 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.380700 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.380799 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.383327 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:24.383332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:24.383335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:24.383395 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:24.386451 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.387221 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.387225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.387227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.387356 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.387726 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.387729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.387730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.388064 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:24.388243 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.388723 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.388926 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:44:24.393478 20194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:44:24.400599 20194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:44:24.429209 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.413792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33053 (local address 127.0.0.1:43996)
0927 21:44:24.414518 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.414573 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:24.414792 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:24.421830 (+ 7038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:24.421843 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:24.421879 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:24.422383 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.422396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.424279 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.424287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.426662 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.426675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.428017 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.428024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.428212 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.428341 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 21:44:24.428508 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 21:44:24.428718 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":41,"thread_start_us":1711,"threads_started":1}
I20260927 21:44:24.430171 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.414385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.0.202.130:33053)
0927 21:44:24.420945 (+ 6560us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.420959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.421002 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:44:24.421142 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.421151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.421487 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.422651 (+ 1164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:24.422893 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.424125 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.426837 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.427803 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.427971 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.428201 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.428560 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.429945 (+ 1385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6331,"thread_start_us":6217,"threads_started":1}
I20260927 21:44:24.431402 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.413837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41877 (local address 127.0.0.1:53976)
0927 21:44:24.421531 (+ 7694us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.421558 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:24.421726 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:24.423099 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:24.423102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:24.423118 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:24.423682 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.423690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.425722 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.425726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.428446 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.428464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.430542 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.430548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.430743 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.430775 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:44:24.430864 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:44:24.430987 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7484,"mutex_wait_us":119,"thread_start_us":7616,"threads_started":1}
I20260927 21:44:24.431692 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.414868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.0.202.129:34427)
0927 21:44:24.415677 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.415689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.421841 (+ 6152us) server_negotiation.cc:407] Connection header received
0927 21:44:24.422217 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.422227 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.422533 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.422716 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:24.423917 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.425277 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.428651 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.430180 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.430299 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.430558 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.431175 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.431348 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":386,"threads_started":1}
I20260927 21:44:24.432356 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.414369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.0.202.131:41877)
0927 21:44:24.422303 (+ 7934us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.422311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.422343 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:44:24.422429 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.422435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.422644 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.422815 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:24.423939 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.425575 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.428671 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.430358 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.430481 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.430724 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.430974 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.431165 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7759,"thread_start_us":8493,"threads_started":1}
I20260927 21:44:24.432411 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.413978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36002)
0927 21:44:24.421715 (+ 7737us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.421731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:24.421964 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:24.423058 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:24.423066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:24.423105 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:24.423677 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.423690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.425811 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.425816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.428481 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.428490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.430767 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.430774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.430945 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.430977 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:44:24.431891 (+ 914us) client_negotiation.cc:240] Negotiation successful
0927 21:44:24.432023 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7624,"mutex_wait_us":212,"thread_start_us":7115,"threads_started":1}
I20260927 21:44:24.434784 20452 tablet_service.cc:1467] Processing CreateTablet for tablet 0451b70eb68c479082195f609df23cf1 (DEFAULT_TABLE table=test-table [id=45fbacdf40d247bf9448897fb17b1570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:24.436252 20452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0451b70eb68c479082195f609df23cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:24.436336 20586 tablet_service.cc:1467] Processing CreateTablet for tablet 0451b70eb68c479082195f609df23cf1 (DEFAULT_TABLE table=test-table [id=45fbacdf40d247bf9448897fb17b1570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:24.436311 20318 tablet_service.cc:1467] Processing CreateTablet for tablet 0451b70eb68c479082195f609df23cf1 (DEFAULT_TABLE table=test-table [id=45fbacdf40d247bf9448897fb17b1570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:24.437935 20318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0451b70eb68c479082195f609df23cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:24.437925 20586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0451b70eb68c479082195f609df23cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:24.448793 20670 tablet_bootstrap.cc:492] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Bootstrap starting.
I20260927 21:44:24.452327 20670 tablet_bootstrap.cc:654] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:24.453644 20672 tablet_bootstrap.cc:492] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Bootstrap starting.
I20260927 21:44:24.453907 20670 log.cc:824] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:24.457070 20671 tablet_bootstrap.cc:492] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Bootstrap starting.
I20260927 21:44:24.457259 20672 tablet_bootstrap.cc:654] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:24.458513 20670 tablet_bootstrap.cc:492] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: No bootstrap required, opened a new log
I20260927 21:44:24.458561 20672 log.cc:824] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:24.458850 20670 ts_tablet_manager.cc:1397] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260927 21:44:24.460517 20671 tablet_bootstrap.cc:654] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:24.461710 20671 log.cc:824] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:24.462446 20672 tablet_bootstrap.cc:492] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: No bootstrap required, opened a new log
I20260927 21:44:24.462718 20672 ts_tablet_manager.cc:1397] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 21:44:24.464483 20671 tablet_bootstrap.cc:492] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: No bootstrap required, opened a new log
I20260927 21:44:24.464725 20671 ts_tablet_manager.cc:1397] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 21:44:24.469427 20670 raft_consensus.cc:348] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.469975 20670 raft_consensus.cc:374] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:24.470077 20670 raft_consensus.cc:729] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47131b4d98134b09ba5dcacee796ae07, State: Initialized, Role: FOLLOWER
I20260927 21:44:24.470770 20670 consensus_queue.cc:260] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.472761 20517 heartbeater.cc:503] Master 127.0.202.190:35841 was elected leader, sending a full tablet report...
I20260927 21:44:24.472633 20671 raft_consensus.cc:348] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.473065 20671 raft_consensus.cc:374] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:24.473193 20671 raft_consensus.cc:729] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7729a85238944feebc4f366fdf0ca909, State: Initialized, Role: FOLLOWER
I20260927 21:44:24.473856 20671 consensus_queue.cc:260] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.474435 20672 raft_consensus.cc:348] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.474848 20672 raft_consensus.cc:374] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:24.474941 20672 raft_consensus.cc:729] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97c7a5e6b2d40c1a657d9cc5d9c4a49, State: Initialized, Role: FOLLOWER
I20260927 21:44:24.475054 20670 ts_tablet_manager.cc:1428] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 21:44:24.475534 20672 consensus_queue.cc:260] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.476073 20651 heartbeater.cc:503] Master 127.0.202.190:35841 was elected leader, sending a full tablet report...
I20260927 21:44:24.476781 20671 ts_tablet_manager.cc:1428] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260927 21:44:24.478888 20672 ts_tablet_manager.cc:1428] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260927 21:44:24.509615 20384 tablet.cc:2369] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:44:24.549949 20518 tablet.cc:2369] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:44:24.592166 20652 tablet.cc:2369] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:24.616317 20676 raft_consensus.cc:482] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:44:24.616606 20676 raft_consensus.cc:504] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.618275 20676 leader_election.cc:288] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7729a85238944feebc4f366fdf0ca909 (127.0.202.131:41877), c97c7a5e6b2d40c1a657d9cc5d9c4a49 (127.0.202.129:34427)
I20260927 21:44:24.628602 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.618742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:39521 (local address 127.0.202.129:34427)
0927 21:44:24.619166 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.619176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.620053 (+ 877us) server_negotiation.cc:407] Connection header received
0927 21:44:24.620376 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.620384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.620469 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.620603 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:24.621333 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.622558 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.625848 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.627137 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.627385 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.627450 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.628109 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.628348 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260927 21:44:24.628572 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.618787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:49125 (local address 127.0.202.131:41877)
0927 21:44:24.618977 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.618991 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.620532 (+ 1541us) server_negotiation.cc:407] Connection header received
0927 21:44:24.620691 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.620696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.620789 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.620867 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:24.621546 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.622701 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.625848 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.627140 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.627348 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.627436 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.628109 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.628261 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:44:24.628726 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.618628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.202.130:39521)
0927 21:44:24.619916 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.619935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:24.620155 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:24.620637 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:24.620642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:24.620671 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:24.621141 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.621159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.622704 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.622708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.625627 (+ 2919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.625640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.627521 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.627528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.627822 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.627914 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 21:44:24.628024 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:44:24.628198 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":165,"thread_start_us":975,"threads_started":1}
I20260927 21:44:24.628840 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.618690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:41877 (local address 127.0.202.130:49125)
0927 21:44:24.620410 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.620425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:24.620539 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:24.620920 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:24.620926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:24.620959 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:24.621390 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.621400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.622848 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.622853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.625622 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:24.625640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.627521 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.627528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:24.627825 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.627914 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 21:44:24.628024 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:44:24.628255 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"mutex_wait_us":341,"thread_start_us":1308,"threads_started":1}
I20260927 21:44:24.630422 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0451b70eb68c479082195f609df23cf1" candidate_uuid: "47131b4d98134b09ba5dcacee796ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7729a85238944feebc4f366fdf0ca909" is_pre_election: true
I20260927 21:44:24.630407 20338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0451b70eb68c479082195f609df23cf1" candidate_uuid: "47131b4d98134b09ba5dcacee796ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" is_pre_election: true
I20260927 21:44:24.631083 20606 raft_consensus.cc:2454] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47131b4d98134b09ba5dcacee796ae07 in term 0.
I20260927 21:44:24.631081 20338 raft_consensus.cc:2454] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47131b4d98134b09ba5dcacee796ae07 in term 0.
I20260927 21:44:24.631996 20407 leader_election.cc:302] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47131b4d98134b09ba5dcacee796ae07, 7729a85238944feebc4f366fdf0ca909; no voters:
I20260927 21:44:24.632884 20676 raft_consensus.cc:2781] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:44:24.633075 20676 raft_consensus.cc:482] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:44:24.633210 20676 raft_consensus.cc:3037] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:24.636965 20676 raft_consensus.cc:504] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.637866 20676 leader_election.cc:288] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [CANDIDATE]: Term 1 election: Requested vote from peers 7729a85238944feebc4f366fdf0ca909 (127.0.202.131:41877), c97c7a5e6b2d40c1a657d9cc5d9c4a49 (127.0.202.129:34427)
I20260927 21:44:24.638444 20606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0451b70eb68c479082195f609df23cf1" candidate_uuid: "47131b4d98134b09ba5dcacee796ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7729a85238944feebc4f366fdf0ca909"
I20260927 21:44:24.638502 20338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0451b70eb68c479082195f609df23cf1" candidate_uuid: "47131b4d98134b09ba5dcacee796ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49"
I20260927 21:44:24.638734 20606 raft_consensus.cc:3037] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:24.638767 20338 raft_consensus.cc:3037] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:24.641723 20338 raft_consensus.cc:2454] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47131b4d98134b09ba5dcacee796ae07 in term 1.
I20260927 21:44:24.641757 20606 raft_consensus.cc:2454] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47131b4d98134b09ba5dcacee796ae07 in term 1.
I20260927 21:44:24.642364 20408 leader_election.cc:302] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47131b4d98134b09ba5dcacee796ae07, c97c7a5e6b2d40c1a657d9cc5d9c4a49; no voters:
I20260927 21:44:24.642777 20676 raft_consensus.cc:2781] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:24.643218 20676 raft_consensus.cc:686] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 1 LEADER]: Becoming Leader. State: Replica: 47131b4d98134b09ba5dcacee796ae07, State: Running, Role: LEADER
I20260927 21:44:24.643771 20676 consensus_queue.cc:237] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } }
I20260927 21:44:24.649546 20192 catalog_manager.cc:5495] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47131b4d98134b09ba5dcacee796ae07 (127.0.202.130). New cstate: current_term: 1 leader_uuid: "47131b4d98134b09ba5dcacee796ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47131b4d98134b09ba5dcacee796ae07" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 } health_report { overall_health: UNKNOWN } } }
I20260927 21:44:24.674340 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.667854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.0.202.130:33053)
0927 21:44:24.668095 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.668101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.668120 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:44:24.668473 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.668477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.668569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.668752 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:44:24.669140 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.669941 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.671856 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.672262 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.672437 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.672970 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:44:24.673492 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:44:24.673675 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.673736 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.673819 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:44:24.682984 20606 raft_consensus.cc:1261] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 1 FOLLOWER]: Refusing update from remote peer 47131b4d98134b09ba5dcacee796ae07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:24.683079 20338 raft_consensus.cc:1261] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 1 FOLLOWER]: Refusing update from remote peer 47131b4d98134b09ba5dcacee796ae07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:24.684077 20681 consensus_queue.cc:1035] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7729a85238944feebc4f366fdf0ca909" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 41877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:24.684485 20676 consensus_queue.cc:1035] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97c7a5e6b2d40c1a657d9cc5d9c4a49" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:24.697778 20686 mvcc.cc:205] Tried to move back new op lower bound from 7334074223331180544 to 7334074223185424384. Current Snapshot: MvccSnapshot[applied={T|T < 7334074223331180544}]
I20260927 21:44:24.697489 20684 mvcc.cc:205] Tried to move back new op lower bound from 7334074223331180544 to 7334074223185424384. Current Snapshot: MvccSnapshot[applied={T|T < 7334074223331180544}]
I20260927 21:44:24.712112 20192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:24.712515 20192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:24.716943 20192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c754c08acc7048f4bab8af8d990be399: Sending DeleteTablet for 3 replicas of tablet 0451b70eb68c479082195f609df23cf1
I20260927 21:44:24.718344 20586 tablet_service.cc:1514] Processing DeleteTablet for tablet 0451b70eb68c479082195f609df23cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:24 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:53976
I20260927 21:44:24.718354 20452 tablet_service.cc:1514] Processing DeleteTablet for tablet 0451b70eb68c479082195f609df23cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:24 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:43996
I20260927 21:44:24.718392 20318 tablet_service.cc:1514] Processing DeleteTablet for tablet 0451b70eb68c479082195f609df23cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:24 UTC) from {username='kudu', principal='kudu/127.0.202.190@KRBTEST.COM'} at 127.0.0.1:36002
I20260927 21:44:24.719448 20693 tablet_replica.cc:330] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: stopping tablet replica
I20260927 21:44:24.720098 20695 tablet_replica.cc:330] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: stopping tablet replica
I20260927 21:44:24.720701 20695 raft_consensus.cc:2229] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:24.721917 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 20255
I20260927 21:44:24.722530 20695 raft_consensus.cc:2258] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:24.721909 20693 raft_consensus.cc:2229] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:44:24.723517 20693 raft_consensus.cc:2258] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:24.726559 20695 ts_tablet_manager.cc:1905] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:24.729257 20693 ts_tablet_manager.cc:1905] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:24.729970 20694 tablet_replica.cc:330] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: stopping tablet replica
I20260927 21:44:24.730547 20694 raft_consensus.cc:2229] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:24.734503 20694 raft_consensus.cc:2258] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:24.737603 20695 ts_tablet_manager.cc:1918] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:24.737787 20695 log.cc:1196] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/0451b70eb68c479082195f609df23cf1
I20260927 21:44:24.738153 20694 ts_tablet_manager.cc:1905] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:24.738348 20695 ts_tablet_manager.cc:1939] T 0451b70eb68c479082195f609df23cf1 P 7729a85238944feebc4f366fdf0ca909: Deleting consensus metadata
I20260927 21:44:24.740439 20180 catalog_manager.cc:4841] TS 7729a85238944feebc4f366fdf0ca909 (127.0.202.131:41877): tablet 0451b70eb68c479082195f609df23cf1 (table test-table [id=45fbacdf40d247bf9448897fb17b1570]) successfully deleted
Sep 27 21:44:24 dist-test-slave-25dq krb5kdc[20149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.129@KRBTEST.COM
I20260927 21:44:24.744277 20693 ts_tablet_manager.cc:1918] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:24.744505 20693 log.cc:1196] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/0451b70eb68c479082195f609df23cf1
I20260927 21:44:24.745059 20693 ts_tablet_manager.cc:1939] T 0451b70eb68c479082195f609df23cf1 P 47131b4d98134b09ba5dcacee796ae07: Deleting consensus metadata
I20260927 21:44:24.746956 20178 catalog_manager.cc:4841] TS 47131b4d98134b09ba5dcacee796ae07 (127.0.202.130:33053): tablet 0451b70eb68c479082195f609df23cf1 (table test-table [id=45fbacdf40d247bf9448897fb17b1570]) successfully deleted
I20260927 21:44:24.750474 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.723489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.0.202.129:34427)
0927 21:44:24.723718 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.723726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.728953 (+ 5227us) server_negotiation.cc:407] Connection header received
0927 21:44:24.729168 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.729174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.729275 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.729401 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:24.731449 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.732763 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.735000 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.735748 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.735861 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.742224 (+ 6363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:24.742244 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:24.742263 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:24.742311 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:24.746497 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.747711 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.747719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.747722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.747878 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.748487 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.748492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.748496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.748956 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:24.749466 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.749715 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.750003 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:44:24.751251 20378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:24.880712 20694 ts_tablet_manager.cc:1918] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:24.881007 20694 log.cc:1196] T 0451b70eb68c479082195f609df23cf1 P c97c7a5e6b2d40c1a657d9cc5d9c4a49: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790545135802652-810-0/minicluster-data/ts-0/wal/wals/0451b70eb68c479082195f609df23cf1
I20260927 21:44:24.881449 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 20255
W20260927 21:44:24.906469 20181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.202.129:34427 (error 108)
I20260927 21:44:24.906636 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 20389
I20260927 21:44:24.908481 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.907907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36010)
0927 21:44:24.908133 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.908260 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 21:44:24.908856 20181 catalog_manager.cc:4584] TS c97c7a5e6b2d40c1a657d9cc5d9c4a49 (127.0.202.129:34427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0451b70eb68c479082195f609df23cf1: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Sep 27 21:44:24 dist-test-slave-25dq krb5kdc[20149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.130@KRBTEST.COM
I20260927 21:44:24.926867 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.908078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.0.202.130:33053)
0927 21:44:24.908332 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:24.908339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:24.908359 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:44:24.908424 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:24.908430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:24.908534 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:24.908640 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:24.910286 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.911531 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.913203 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:24.913894 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:24.914031 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:24.919106 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:24.919123 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:24.919147 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:24.919207 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:24.923684 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.924604 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.924615 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.924618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.924806 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:24.925184 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:24.925189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:24.925193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:24.925664 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:24.926049 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:24.926229 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:44:24.926458 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:44:24.927848 20512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:24.962855 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.962262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36026)
0927 21:44:24.962521 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.962633 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 21:44:24.964030 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:24.963605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36032)
0927 21:44:24.963759 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:24.963895 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 21:44:25.024720 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:25.024126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36044)
0927 21:44:25.024431 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:25.024534 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 21:44:25.026013 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:25.025605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36050)
0927 21:44:25.025814 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:25.025891 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 21:44:25.046576 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 20389
I20260927 21:44:25.065788 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 20523
Sep 27 21:44:25 dist-test-slave-25dq krb5kdc[20149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.131@KRBTEST.COM
I20260927 21:44:25.082176 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:25.066600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.202.131:41877)
0927 21:44:25.066818 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:25.066825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:25.066866 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:44:25.067090 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:25.067100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:25.067168 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:25.067298 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:25.068757 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:25.069462 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:25.070760 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:25.071155 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:25.071291 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:25.076148 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:25.076165 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:25.076182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:25.076227 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:25.079184 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:25.079961 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:25.079968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:25.079971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:25.080116 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:25.080577 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:25.080582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:25.080585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:25.080958 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:25.081268 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:25.081609 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 21:44:25.081838 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:44:25.082808 20646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:25.106920 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:25.106367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36052)
0927 21:44:25.106598 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:25.106707 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 21:44:25.108028 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:25.107624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:34427 (local address 127.0.0.1:36058)
0927 21:44:25.107842 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:25.107916 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:34427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 21:44:25.175608 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 20523
I20260927 21:44:25.192626 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 20162
I20260927 21:44:25.193828 20224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:25.308336 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 20162
2026-09-27T21:44:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4587 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T21:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:25Z Disabled control of system clock
I20260927 21:44:25.388015 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35827
--webserver_interface=127.0.202.190
--webserver_port=0
--builtin_ntp_servers=127.0.202.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:44:25.731297 20705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:25.731739 20705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:25.731912 20705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:25.740414 20705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:25.740590 20705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:25.740685 20705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:25.740797 20705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:25.740890 20705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:25.751180 20705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:35827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:25.752781 20705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:25.754693 20705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:25.767171 20711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:25.767277 20710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:25.767184 20713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:25.767970 20705 server_base.cc:1034] running on GCE node
I20260927 21:44:25.768852 20705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:25.770330 20705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:25.771533 20705 hybrid_clock.cc:648] HybridClock initialized: now 1790545465771492 us; error 28 us; skew 500 ppm
I20260927 21:44:25.771916 20705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:25.774266 20705 webserver.cc:460] Webserver started at http://127.0.202.190:39867/ using document root <none> and password file <none>
I20260927 21:44:25.774993 20705 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:25.775081 20705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:25.775458 20705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:25.778661 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "c3f79d87d447484dae67f4d291867c35"
format_stamp: "Formatted at 2026-09-27 21:44:25 on dist-test-slave-25dq"
server_key: "e51d6eeedc851630e8a848a908750982"
server_key_iv: "e3db92e88d432526556e8ace19dff2ea"
server_key_version: "encryptionkey@0"
I20260927 21:44:25.779541 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "c3f79d87d447484dae67f4d291867c35"
format_stamp: "Formatted at 2026-09-27 21:44:25 on dist-test-slave-25dq"
server_key: "e51d6eeedc851630e8a848a908750982"
server_key_iv: "e3db92e88d432526556e8ace19dff2ea"
server_key_version: "encryptionkey@0"
I20260927 21:44:25.785262 20705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:44:25.789271 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:25.790792 20705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:25.790992 20705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "c3f79d87d447484dae67f4d291867c35"
format_stamp: "Formatted at 2026-09-27 21:44:25 on dist-test-slave-25dq"
server_key: "e51d6eeedc851630e8a848a908750982"
server_key_iv: "e3db92e88d432526556e8ace19dff2ea"
server_key_version: "encryptionkey@0"
I20260927 21:44:25.791214 20705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:25.810905 20705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:25.812014 20705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:25.812394 20705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:25.831983 20705 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:35827
I20260927 21:44:25.832002 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:35827 every 8 connection(s)
I20260927 21:44:25.833431 20705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:25.837306 20771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:25.843511 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 20705
I20260927 21:44:25.843715 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:25.844261 810 external_mini_cluster.cc:1428] Setting key cf3744c4f6af3c1ac2826283225f23a8
I20260927 21:44:25.849073 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35: Bootstrap starting.
I20260927 21:44:25.852196 20771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:25.853495 20771 log.cc:824] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:25.856992 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35: No bootstrap required, opened a new log
I20260927 21:44:25.859380 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:25.846406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.0.202.190:35827)
0927 21:44:25.847409 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:25.847421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:25.847462 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:44:25.849325 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:25.849348 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:25.849839 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:25.850287 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:25.850894 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:25.852213 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:25.854207 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:25.855331 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:25.855490 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:25.856882 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:25.856889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:25.856904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:25.857580 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:25.857667 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:25.857697 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:25.857797 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:25.858019 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 21:44:25.858364 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":335,"threads_started":1}
I20260927 21:44:25.864920 20771 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f79d87d447484dae67f4d291867c35" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35827 } }
I20260927 21:44:25.865331 20771 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:25.865475 20771 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f79d87d447484dae67f4d291867c35, State: Initialized, Role: FOLLOWER
I20260927 21:44:25.866039 20771 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f79d87d447484dae67f4d291867c35" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35827 } }
I20260927 21:44:25.866282 20771 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:25.866408 20771 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:25.866601 20771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:25.868932 20771 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f79d87d447484dae67f4d291867c35" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35827 } }
I20260927 21:44:25.869457 20771 leader_election.cc:302] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f79d87d447484dae67f4d291867c35; no voters:
I20260927 21:44:25.870088 20771 leader_election.cc:288] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:25.870302 20776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:25.871083 20776 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [term 1 LEADER]: Becoming Leader. State: Replica: c3f79d87d447484dae67f4d291867c35, State: Running, Role: LEADER
I20260927 21:44:25.871627 20776 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f79d87d447484dae67f4d291867c35" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35827 } }
I20260927 21:44:25.872273 20771 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:25.875532 20778 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f79d87d447484dae67f4d291867c35. Latest consensus state: current_term: 1 leader_uuid: "c3f79d87d447484dae67f4d291867c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f79d87d447484dae67f4d291867c35" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35827 } } }
I20260927 21:44:25.876011 20778 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:25.876718 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f79d87d447484dae67f4d291867c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f79d87d447484dae67f4d291867c35" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 35827 } } }
I20260927 21:44:25.877102 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:25.880703 20786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:25.887301 20786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:25.897158 20786 catalog_manager.cc:1347] Generated new cluster ID: bd3f5892696a400aaef2397e42ea9352
I20260927 21:44:25.897339 20786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:25.909673 20786 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:25.911628 20786 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:25.919351 20786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35: Generated new TSK 0
I20260927 21:44:25.920434 20786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:25.926530 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.129:0
--local_ip_for_outbound_sockets=127.0.202.129
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35827
--builtin_ntp_servers=127.0.202.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:44:26.285372 20795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:26.285727 20795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:26.285897 20795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:26.293510 20795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:26.293664 20795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:26.293841 20795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.129
I20260927 21:44:26.303398 20795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.202.129
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.129
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:26.304879 20795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:26.306691 20795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:26.320081 20803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:26.320129 20801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:26.320047 20800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:26.325920 20795 server_base.cc:1034] running on GCE node
I20260927 21:44:26.326651 20795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:26.327710 20795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:26.328933 20795 hybrid_clock.cc:648] HybridClock initialized: now 1790545466328861 us; error 58 us; skew 500 ppm
I20260927 21:44:26.329350 20795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:26.331809 20795 webserver.cc:460] Webserver started at http://127.0.202.129:36529/ using document root <none> and password file <none>
I20260927 21:44:26.332710 20795 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:26.332806 20795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:26.333161 20795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:26.336185 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data/instance:
uuid: "a20fdac163eb47eabbe433a52d7b2645"
format_stamp: "Formatted at 2026-09-27 21:44:26 on dist-test-slave-25dq"
server_key: "38279d51661aa213fbe432321d54a5f4"
server_key_iv: "3b0c7281155e8f6ec00afbb7a915a0df"
server_key_version: "encryptionkey@0"
I20260927 21:44:26.337062 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/wal/instance:
uuid: "a20fdac163eb47eabbe433a52d7b2645"
format_stamp: "Formatted at 2026-09-27 21:44:26 on dist-test-slave-25dq"
server_key: "38279d51661aa213fbe432321d54a5f4"
server_key_iv: "3b0c7281155e8f6ec00afbb7a915a0df"
server_key_version: "encryptionkey@0"
I20260927 21:44:26.342401 20795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:44:26.346082 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:26.347498 20795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:44:26.347672 20795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/wal
uuid: "a20fdac163eb47eabbe433a52d7b2645"
format_stamp: "Formatted at 2026-09-27 21:44:26 on dist-test-slave-25dq"
server_key: "38279d51661aa213fbe432321d54a5f4"
server_key_iv: "3b0c7281155e8f6ec00afbb7a915a0df"
server_key_version: "encryptionkey@0"
I20260927 21:44:26.347913 20795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:26.380762 20795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:26.381666 20795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:26.381978 20795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:26.382975 20795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:26.384627 20795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:26.384744 20795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:26.384886 20795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:26.384974 20795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:26.419145 20795 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.129:35175
I20260927 21:44:26.419200 20921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.129:35175 every 8 connection(s)
I20260927 21:44:26.420648 20795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/data/info.pb
I20260927 21:44:26.426172 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 20795
I20260927 21:44:26.426337 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-0/wal/instance
I20260927 21:44:26.426788 810 external_mini_cluster.cc:1428] Setting key 120db77b4c308839d1ce1818377e8fde
I20260927 21:44:26.429716 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.130:0
--local_ip_for_outbound_sockets=127.0.202.130
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35827
--builtin_ntp_servers=127.0.202.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:44:26.435107 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:26.423195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:58401 (local address 127.0.202.190:35827)
0927 21:44:26.423702 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:26.423709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:26.425027 (+ 1318us) server_negotiation.cc:407] Connection header received
0927 21:44:26.427116 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:26.427121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:26.427218 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:26.427353 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:26.428682 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.429495 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.432106 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.432573 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.432676 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:26.434372 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:26.434377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:26.434380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:26.434399 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:26.434439 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:26.434451 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:26.434521 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:26.434778 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 21:44:26.434858 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":218,"threads_started":1}
I20260927 21:44:26.436725 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:26.423834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35827 (local address 127.0.202.129:58401)
0927 21:44:26.424872 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:26.424917 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:26.426791 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:26.427623 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:26.427637 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:26.427769 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:26.428452 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:26.428493 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.429650 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.429655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:26.431948 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:26.431966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.432823 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.432833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:26.433166 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:26.434000 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:26.434005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:26.434063 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:26.434166 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:26.434551 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:26.434565 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:26.434576 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:44:26.434763 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 21:44:26.435222 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":419,"threads_started":1}
I20260927 21:44:26.438349 20922 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35827
I20260927 21:44:26.438706 20922 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:26.439563 20922 heartbeater.cc:511] Master 127.0.202.190:35827 requested a full tablet report, sending...
I20260927 21:44:26.442039 20736 ts_manager.cc:194] Registered new tserver with Master: a20fdac163eb47eabbe433a52d7b2645 (127.0.202.129:35175)
I20260927 21:44:26.444276 20736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.129:58401
W20260927 21:44:26.775817 20926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:26.776103 20926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:26.776224 20926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:26.783557 20926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:26.783672 20926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:26.783787 20926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.130
I20260927 21:44:26.793316 20926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.202.130
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.130
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:26.794600 20926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:26.796448 20926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:26.810283 20931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:26.810235 20932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:26.810388 20934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:26.812131 20926 server_base.cc:1034] running on GCE node
I20260927 21:44:26.813035 20926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:26.814373 20926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:26.815653 20926 hybrid_clock.cc:648] HybridClock initialized: now 1790545466815584 us; error 48 us; skew 500 ppm
I20260927 21:44:26.816146 20926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:26.819391 20926 webserver.cc:460] Webserver started at http://127.0.202.130:45857/ using document root <none> and password file <none>
I20260927 21:44:26.820420 20926 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:26.820551 20926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:26.820964 20926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:26.823866 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data/instance:
uuid: "18b9253ad9b342e0aede019964b99490"
format_stamp: "Formatted at 2026-09-27 21:44:26 on dist-test-slave-25dq"
server_key: "442d580777963cbf8a40eed96c421455"
server_key_iv: "2534623187a01a136c3b445426b54f24"
server_key_version: "encryptionkey@0"
I20260927 21:44:26.824678 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal/instance:
uuid: "18b9253ad9b342e0aede019964b99490"
format_stamp: "Formatted at 2026-09-27 21:44:26 on dist-test-slave-25dq"
server_key: "442d580777963cbf8a40eed96c421455"
server_key_iv: "2534623187a01a136c3b445426b54f24"
server_key_version: "encryptionkey@0"
I20260927 21:44:26.829864 20926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:44:26.833909 20940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:26.835551 20926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:26.835711 20926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal
uuid: "18b9253ad9b342e0aede019964b99490"
format_stamp: "Formatted at 2026-09-27 21:44:26 on dist-test-slave-25dq"
server_key: "442d580777963cbf8a40eed96c421455"
server_key_iv: "2534623187a01a136c3b445426b54f24"
server_key_version: "encryptionkey@0"
I20260927 21:44:26.835953 20926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:26.856153 20926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:26.857438 20926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:26.857774 20926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:26.858783 20926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:26.860471 20926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:26.860580 20926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:26.860692 20926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:26.860812 20926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:26.896727 20926 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.130:33977
I20260927 21:44:26.896771 21052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.130:33977 every 8 connection(s)
I20260927 21:44:26.898070 20926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/data/info.pb
I20260927 21:44:26.901028 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 20926
I20260927 21:44:26.901232 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal/instance
I20260927 21:44:26.901738 810 external_mini_cluster.cc:1428] Setting key 6e07722d5dbc1695a06ac4f346683e7f
I20260927 21:44:26.904763 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.202.131:0
--local_ip_for_outbound_sockets=127.0.202.131
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35827
--builtin_ntp_servers=127.0.202.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:44:26.912680 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:26.900797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:40611 (local address 127.0.202.190:35827)
0927 21:44:26.901041 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:26.901049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:26.902774 (+ 1725us) server_negotiation.cc:407] Connection header received
0927 21:44:26.905100 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:26.905108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:26.905201 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:26.905313 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:26.906473 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.907620 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.909271 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.910034 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.910177 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:26.911824 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:26.911833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:26.911840 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:26.911868 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:26.911912 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:26.911930 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:26.912023 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:26.912334 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:44:26.912444 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 21:44:26.914291 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:26.901423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35827 (local address 127.0.202.130:40611)
0927 21:44:26.902615 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:26.902663 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:26.904737 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:26.905520 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:26.905528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:26.905623 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:26.906285 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:26.906308 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.907796 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.907800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:26.909116 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:26.909128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:26.910303 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:26.910310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:26.910563 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:26.911458 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:26.911461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:26.911506 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:26.911608 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:26.912054 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:26.912064 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:26.912070 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:44:26.912356 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 21:44:26.912945 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":398,"threads_started":1}
I20260927 21:44:26.916092 21053 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35827
I20260927 21:44:26.916529 21053 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:26.917408 21053 heartbeater.cc:511] Master 127.0.202.190:35827 requested a full tablet report, sending...
I20260927 21:44:26.919095 20736 ts_manager.cc:194] Registered new tserver with Master: 18b9253ad9b342e0aede019964b99490 (127.0.202.130:33977)
I20260927 21:44:26.920488 20736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.130:40611
W20260927 21:44:27.244603 21056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:27.244895 21056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:27.244982 21056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:27.252518 21056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:27.252624 21056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:27.252732 21056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.202.131
I20260927 21:44:27.262166 21056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.202.131
--webserver_port=0
--tserver_master_addrs=127.0.202.190:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.202.131
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:27.263432 21056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:27.265202 21056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:27.279297 21061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:27.279286 21064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:27.279277 21062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:27.280131 21056 server_base.cc:1034] running on GCE node
I20260927 21:44:27.281095 21056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:27.282296 21056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:27.283548 21056 hybrid_clock.cc:648] HybridClock initialized: now 1790545467283477 us; error 59 us; skew 500 ppm
I20260927 21:44:27.283938 21056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:44:27.286504 21056 webserver.cc:460] Webserver started at http://127.0.202.131:36655/ using document root <none> and password file <none>
I20260927 21:44:27.287241 21056 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:27.287328 21056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:27.287698 21056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:27.290748 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data/instance:
uuid: "8574c6ef52ea4a17a3ed24f39292c5a2"
format_stamp: "Formatted at 2026-09-27 21:44:27 on dist-test-slave-25dq"
server_key: "37549f1b4365c67dee74fd57ce4b31e0"
server_key_iv: "c91d7f0f2a80e3be40fe0e86caf041c3"
server_key_version: "encryptionkey@0"
I20260927 21:44:27.291543 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal/instance:
uuid: "8574c6ef52ea4a17a3ed24f39292c5a2"
format_stamp: "Formatted at 2026-09-27 21:44:27 on dist-test-slave-25dq"
server_key: "37549f1b4365c67dee74fd57ce4b31e0"
server_key_iv: "c91d7f0f2a80e3be40fe0e86caf041c3"
server_key_version: "encryptionkey@0"
I20260927 21:44:27.297049 21056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 21:44:27.300753 21070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:27.302234 21056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:44:27.302400 21056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal
uuid: "8574c6ef52ea4a17a3ed24f39292c5a2"
format_stamp: "Formatted at 2026-09-27 21:44:27 on dist-test-slave-25dq"
server_key: "37549f1b4365c67dee74fd57ce4b31e0"
server_key_iv: "c91d7f0f2a80e3be40fe0e86caf041c3"
server_key_version: "encryptionkey@0"
I20260927 21:44:27.302636 21056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:27.325075 21056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:27.325978 21056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:27.326289 21056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:27.327312 21056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:44:27.328929 21056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:44:27.329021 21056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:27.329169 21056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:44:27.329259 21056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:27.363897 21056 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.131:36247
I20260927 21:44:27.364009 21182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.131:36247 every 8 connection(s)
I20260927 21:44:27.365258 21056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/data/info.pb
I20260927 21:44:27.373924 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21056
I20260927 21:44:27.374114 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal/instance
I20260927 21:44:27.374722 810 external_mini_cluster.cc:1428] Setting key 1d7eb531694fec57c45ed77de4611bca
I20260927 21:44:27.379797 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.368342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.131:49957 (local address 127.0.202.190:35827)
0927 21:44:27.368554 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.368559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.370044 (+ 1485us) server_negotiation.cc:407] Connection header received
0927 21:44:27.371862 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.371867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.371955 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.372072 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:27.373263 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.374567 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.376725 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.377284 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.377413 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.378977 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:27.378982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:27.378985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:27.379003 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:27.379034 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:27.379046 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:27.379113 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.379332 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.379410 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:44:27.381160 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.368844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:35827 (local address 127.0.202.131:49957)
0927 21:44:27.369862 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.369900 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.371565 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.372360 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.372379 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.372476 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:44:27.373110 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.373127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.374763 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.374767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.376570 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.376583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.377516 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.377522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.377854 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.378644 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:44:27.378647 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:44:27.378691 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:44:27.378784 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:44:27.379133 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:44:27.379141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:44:27.379147 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.379303 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.379827 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":347,"threads_started":1}
I20260927 21:44:27.382745 21183 heartbeater.cc:347] Connected to a master server at 127.0.202.190:35827
I20260927 21:44:27.383188 21183 heartbeater.cc:464] Registering TS with master...
I20260927 21:44:27.384116 21183 heartbeater.cc:511] Master 127.0.202.190:35827 requested a full tablet report, sending...
I20260927 21:44:27.385848 20736 ts_manager.cc:194] Registered new tserver with Master: 8574c6ef52ea4a17a3ed24f39292c5a2 (127.0.202.131:36247)
I20260927 21:44:27.386777 20736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.202.131:49957
I20260927 21:44:27.390978 810 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:44:27.402035 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.395787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.0.202.190:35827)
0927 21:44:27.396043 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.396048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.396528 (+ 480us) server_negotiation.cc:407] Connection header received
0927 21:44:27.396670 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.396675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.396764 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.396867 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:27.397405 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.398103 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.399887 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.400357 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.400467 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.401450 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:27.401455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:27.401458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:27.401475 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:27.401505 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:27.401518 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:27.401585 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.401727 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.401803 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:44:27.406275 20736 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:44:27.413105 20736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:44:27.437832 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.427668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33977 (local address 127.0.0.1:33944)
0927 21:44:27.428893 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.428911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.429018 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.430313 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.430321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.430349 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:27.430884 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.430903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.432841 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.432847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.435272 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.435297 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.436728 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.436734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.437026 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.437070 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.437233 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.437446 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":112,"thread_start_us":2988,"threads_started":1}
I20260927 21:44:27.438905 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.428428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.0.202.130:33977)
0927 21:44:27.429802 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.429808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.429831 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:44:27.429916 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.429921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.430102 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.430306 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:27.431156 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.432661 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.435529 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.436520 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.436678 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.436940 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.438350 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.438531 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":1116,"threads_started":1}
I20260927 21:44:27.439734 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.427044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:36247 (local address 127.0.0.1:37036)
0927 21:44:27.427992 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.428028 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.428200 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.431830 (+ 3630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.431837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.431859 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:27.432286 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.432339 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.434248 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.434254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.436991 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.437007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.438923 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.438933 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.439125 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.439163 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.439231 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.439353 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":6,"thread_start_us":546,"threads_started":1}
I20260927 21:44:27.440456 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.428399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53046)
0927 21:44:27.431676 (+ 3277us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.431695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.431852 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.432567 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.432572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.432602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:27.433039 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.433051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.434733 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.434744 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.437242 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.437254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.439547 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.439554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.439777 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.439818 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.439909 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.440040 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3140,"mutex_wait_us":46,"thread_start_us":745,"threads_started":1}
I20260927 21:44:27.441025 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.427503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.202.131:36247)
0927 21:44:27.431227 (+ 3724us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.431236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.431273 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:44:27.431377 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.431385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.431635 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.431820 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:27.432500 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.434093 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.437227 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.438700 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.440047 (+ 1347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.440374 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.440441 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.440614 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3555,"thread_start_us":3398,"threads_started":1}
I20260927 21:44:27.441363 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.428954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.202.129:35175)
0927 21:44:27.429485 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.429496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.432003 (+ 2507us) server_negotiation.cc:407] Connection header received
0927 21:44:27.432120 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.432130 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.432396 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.432614 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:27.433244 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.434539 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.437431 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.438933 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.439087 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.439401 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.440778 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.440938 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":321,"threads_started":1}
I20260927 21:44:27.442982 20988 tablet_service.cc:1467] Processing CreateTablet for tablet a4e477e6f0b04a99a44d2e2b4521d506 (DEFAULT_TABLE table=test-table [id=1b67be01d379482f99c04d0d986012d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:27.444440 20988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e477e6f0b04a99a44d2e2b4521d506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:27.444468 21118 tablet_service.cc:1467] Processing CreateTablet for tablet a4e477e6f0b04a99a44d2e2b4521d506 (DEFAULT_TABLE table=test-table [id=1b67be01d379482f99c04d0d986012d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:27.445932 21118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e477e6f0b04a99a44d2e2b4521d506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:27.446097 20857 tablet_service.cc:1467] Processing CreateTablet for tablet a4e477e6f0b04a99a44d2e2b4521d506 (DEFAULT_TABLE table=test-table [id=1b67be01d379482f99c04d0d986012d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:44:27.447417 20857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e477e6f0b04a99a44d2e2b4521d506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:27.447788 20922 heartbeater.cc:503] Master 127.0.202.190:35827 was elected leader, sending a full tablet report...
I20260927 21:44:27.460424 21201 tablet_bootstrap.cc:492] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Bootstrap starting.
I20260927 21:44:27.463142 21203 tablet_bootstrap.cc:492] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: Bootstrap starting.
I20260927 21:44:27.464318 21202 tablet_bootstrap.cc:492] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Bootstrap starting.
I20260927 21:44:27.465345 21201 tablet_bootstrap.cc:654] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:27.466779 21201 log.cc:824] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:27.467069 21203 tablet_bootstrap.cc:654] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:27.467216 21202 tablet_bootstrap.cc:654] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:27.468343 21202 log.cc:824] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:27.468479 21203 log.cc:824] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:27.471400 21203 tablet_bootstrap.cc:492] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: No bootstrap required, opened a new log
I20260927 21:44:27.471683 21203 ts_tablet_manager.cc:1397] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 21:44:27.471675 21202 tablet_bootstrap.cc:492] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: No bootstrap required, opened a new log
I20260927 21:44:27.471985 21202 ts_tablet_manager.cc:1397] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 21:44:27.472604 21201 tablet_bootstrap.cc:492] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: No bootstrap required, opened a new log
I20260927 21:44:27.472945 21201 ts_tablet_manager.cc:1397] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260927 21:44:27.479686 21203 raft_consensus.cc:348] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.480232 21203 raft_consensus.cc:374] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:27.480388 21203 raft_consensus.cc:729] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20fdac163eb47eabbe433a52d7b2645, State: Initialized, Role: FOLLOWER
I20260927 21:44:27.481045 21203 consensus_queue.cc:260] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.482792 21203 ts_tablet_manager.cc:1428] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260927 21:44:27.483870 21202 raft_consensus.cc:348] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.484416 21202 raft_consensus.cc:374] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:27.484529 21202 raft_consensus.cc:729] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b9253ad9b342e0aede019964b99490, State: Initialized, Role: FOLLOWER
I20260927 21:44:27.485278 21202 consensus_queue.cc:260] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.486521 21053 heartbeater.cc:503] Master 127.0.202.190:35827 was elected leader, sending a full tablet report...
I20260927 21:44:27.487138 21201 raft_consensus.cc:348] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.487629 21201 raft_consensus.cc:374] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:27.487738 21201 raft_consensus.cc:729] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8574c6ef52ea4a17a3ed24f39292c5a2, State: Initialized, Role: FOLLOWER
I20260927 21:44:27.488585 21201 consensus_queue.cc:260] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.489713 21183 heartbeater.cc:503] Master 127.0.202.190:35827 was elected leader, sending a full tablet report...
I20260927 21:44:27.490648 21202 ts_tablet_manager.cc:1428] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260927 21:44:27.490648 21201 ts_tablet_manager.cc:1428] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
W20260927 21:44:27.617154 21184 tablet.cc:2369] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:27.626690 21207 raft_consensus.cc:482] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:44:27.626996 21207 raft_consensus.cc:504] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.628785 21207 leader_election.cc:288] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8574c6ef52ea4a17a3ed24f39292c5a2 (127.0.202.131:36247), 18b9253ad9b342e0aede019964b99490 (127.0.202.130:33977)
I20260927 21:44:27.635919 21208 raft_consensus.cc:482] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:44:27.636504 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.629234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:47707 (local address 127.0.202.131:36247)
0927 21:44:27.629420 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.629425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.630324 (+ 899us) server_negotiation.cc:407] Connection header received
0927 21:44:27.630567 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.630572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.630640 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.630739 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:27.631377 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.632471 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.634715 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.635516 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.635701 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.635773 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.636089 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.636226 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:44:27.636243 21208 raft_consensus.cc:504] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.636715 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.629431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.129:40691 (local address 127.0.202.130:33977)
0927 21:44:27.629659 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.629665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.630941 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 21:44:27.631106 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.631111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.631211 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.631311 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:27.631977 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.632939 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.634799 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.635609 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.635761 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.635834 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.636307 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.636442 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:44:27.636914 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.629333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:36247 (local address 127.0.202.129:47707)
0927 21:44:27.630209 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.630225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.630417 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.630761 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.630766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.630803 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:27.631213 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.631225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.632606 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.632612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.634497 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.634516 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.635704 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.635710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.635893 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.635943 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.636034 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.636323 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":85,"thread_start_us":462,"threads_started":1}
I20260927 21:44:27.637367 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.629793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.130:33977 (local address 127.0.202.129:40691)
0927 21:44:27.630814 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.630830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.630976 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.631374 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.631379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.631406 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:27.631816 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.631829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.633107 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.633111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.634571 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.634583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.635905 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.635911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.636095 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.636131 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.636955 (+ 824us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.637100 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":65,"thread_start_us":662,"threads_started":1}
I20260927 21:44:27.638072 21138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e477e6f0b04a99a44d2e2b4521d506" candidate_uuid: "a20fdac163eb47eabbe433a52d7b2645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" is_pre_election: true
I20260927 21:44:27.638453 21208 leader_election.cc:288] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8574c6ef52ea4a17a3ed24f39292c5a2 (127.0.202.131:36247), a20fdac163eb47eabbe433a52d7b2645 (127.0.202.129:35175)
I20260927 21:44:27.638353 21008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e477e6f0b04a99a44d2e2b4521d506" candidate_uuid: "a20fdac163eb47eabbe433a52d7b2645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b9253ad9b342e0aede019964b99490" is_pre_election: true
I20260927 21:44:27.638584 21138 raft_consensus.cc:2454] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20fdac163eb47eabbe433a52d7b2645 in term 0.
I20260927 21:44:27.638893 21008 raft_consensus.cc:2454] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20fdac163eb47eabbe433a52d7b2645 in term 0.
I20260927 21:44:27.639595 20813 leader_election.cc:302] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b9253ad9b342e0aede019964b99490, a20fdac163eb47eabbe433a52d7b2645; no voters:
I20260927 21:44:27.640275 21207 raft_consensus.cc:2781] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:44:27.640461 21207 raft_consensus.cc:482] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:44:27.640584 21207 raft_consensus.cc:3037] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:27.643460 21207 raft_consensus.cc:504] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.644717 21008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e477e6f0b04a99a44d2e2b4521d506" candidate_uuid: "a20fdac163eb47eabbe433a52d7b2645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b9253ad9b342e0aede019964b99490"
I20260927 21:44:27.644716 21138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e477e6f0b04a99a44d2e2b4521d506" candidate_uuid: "a20fdac163eb47eabbe433a52d7b2645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2"
I20260927 21:44:27.644930 21008 raft_consensus.cc:3037] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:27.644950 21138 raft_consensus.cc:3037] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:27.647291 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.638957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:40899 (local address 127.0.202.131:36247)
0927 21:44:27.639152 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.639157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.639911 (+ 754us) server_negotiation.cc:407] Connection header received
0927 21:44:27.639976 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.639981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.640067 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.640155 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:27.641071 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.642162 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.644752 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.646001 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.646142 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.646292 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.646846 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.646991 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:44:27.647964 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.638809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.131:36247 (local address 127.0.202.130:40899)
0927 21:44:27.639676 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.639695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.639822 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.640338 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.640346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.640385 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:27.640915 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.640929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.642273 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.642280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.644522 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.644538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.646470 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.646480 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.646654 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.646691 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.647451 (+ 760us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.647596 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":45,"thread_start_us":574,"threads_started":1}
I20260927 21:44:27.648274 21008 raft_consensus.cc:2454] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20fdac163eb47eabbe433a52d7b2645 in term 1.
I20260927 21:44:27.648353 21138 raft_consensus.cc:2454] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20fdac163eb47eabbe433a52d7b2645 in term 1.
I20260927 21:44:27.648490 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.640153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.202.130:42201 (local address 127.0.202.129:35175)
0927 21:44:27.640411 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.640416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.641300 (+ 884us) server_negotiation.cc:407] Connection header received
0927 21:44:27.641510 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.641516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.641610 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.641711 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:44:27.642395 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.643513 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.646104 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.647286 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.647494 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.647561 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.648114 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.648254 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 21:44:27.648757 20813 leader_election.cc:302] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b9253ad9b342e0aede019964b99490, a20fdac163eb47eabbe433a52d7b2645; no voters:
I20260927 21:44:27.648813 21137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e477e6f0b04a99a44d2e2b4521d506" candidate_uuid: "18b9253ad9b342e0aede019964b99490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" is_pre_election: true
I20260927 21:44:27.649204 21137 raft_consensus.cc:2379] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18b9253ad9b342e0aede019964b99490 in current term 1: Already voted for candidate a20fdac163eb47eabbe433a52d7b2645 in this term.
I20260927 21:44:27.649252 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.639842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.202.130:42201)
0927 21:44:27.641179 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.641203 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:44:27.641323 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:44:27.641739 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:44:27.641744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:44:27.641772 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:44:27.642164 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.642176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.643661 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.643667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.645868 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:44:27.645891 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.647719 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.647725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:44:27.647903 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.647945 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:44:27.648704 (+ 759us) client_negotiation.cc:240] Negotiation successful
0927 21:44:27.648857 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":4,"thread_start_us":915,"threads_started":1}
I20260927 21:44:27.649344 21207 leader_election.cc:288] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [CANDIDATE]: Term 1 election: Requested vote from peers 8574c6ef52ea4a17a3ed24f39292c5a2 (127.0.202.131:36247), 18b9253ad9b342e0aede019964b99490 (127.0.202.130:33977)
I20260927 21:44:27.649463 21214 raft_consensus.cc:2781] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:27.649684 21214 raft_consensus.cc:686] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 1 LEADER]: Becoming Leader. State: Replica: a20fdac163eb47eabbe433a52d7b2645, State: Running, Role: LEADER
W20260927 21:44:27.650107 21054 tablet.cc:2369] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:27.650175 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e477e6f0b04a99a44d2e2b4521d506" candidate_uuid: "18b9253ad9b342e0aede019964b99490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20fdac163eb47eabbe433a52d7b2645" is_pre_election: true
I20260927 21:44:27.651194 21214 consensus_queue.cc:237] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } }
I20260927 21:44:27.655619 20943 leader_election.cc:302] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18b9253ad9b342e0aede019964b99490; no voters: 8574c6ef52ea4a17a3ed24f39292c5a2, a20fdac163eb47eabbe433a52d7b2645
I20260927 21:44:27.656466 21208 raft_consensus.cc:2726] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 21:44:27.656962 20735 catalog_manager.cc:5495] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20fdac163eb47eabbe433a52d7b2645 (127.0.202.129). New cstate: current_term: 1 leader_uuid: "a20fdac163eb47eabbe433a52d7b2645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20fdac163eb47eabbe433a52d7b2645" member_type: VOTER last_known_addr { host: "127.0.202.129" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 } health_report { overall_health: UNKNOWN } } }
W20260927 21:44:27.673166 20923 tablet.cc:2369] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:44:27.680470 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.674441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.0.202.129:35175)
0927 21:44:27.674615 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.674620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.674751 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:44:27.674932 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.674936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.675012 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.675138 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:44:27.675670 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.676512 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.678021 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.678548 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.678662 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.679138 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:44:27.679756 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:44:27.679882 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.680002 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.680077 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:44:27.689225 21138 raft_consensus.cc:1261] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 1 FOLLOWER]: Refusing update from remote peer a20fdac163eb47eabbe433a52d7b2645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:27.689525 21008 raft_consensus.cc:1261] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 1 FOLLOWER]: Refusing update from remote peer a20fdac163eb47eabbe433a52d7b2645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:44:27.690280 21214 consensus_queue.cc:1035] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8574c6ef52ea4a17a3ed24f39292c5a2" member_type: VOTER last_known_addr { host: "127.0.202.131" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:27.690624 21207 consensus_queue.cc:1035] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b9253ad9b342e0aede019964b99490" member_type: VOTER last_known_addr { host: "127.0.202.130" port: 33977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:44:27.702893 21217 mvcc.cc:205] Tried to move back new op lower bound from 7334074235644796928 to 7334074235503235072. Current Snapshot: MvccSnapshot[applied={T|T < 7334074235644796928}]
I20260927 21:44:27.706204 21218 mvcc.cc:205] Tried to move back new op lower bound from 7334074235644796928 to 7334074235503235072. Current Snapshot: MvccSnapshot[applied={T|T < 7334074235644796928}]
I20260927 21:44:27.718385 20736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:27.718719 20736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:44:27.722721 20736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f79d87d447484dae67f4d291867c35: Sending DeleteTablet for 3 replicas of tablet a4e477e6f0b04a99a44d2e2b4521d506
I20260927 21:44:27.724057 20857 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e477e6f0b04a99a44d2e2b4521d506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:27 UTC) from {username='slave'} at 127.0.0.1:53046
I20260927 21:44:27.724220 20988 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e477e6f0b04a99a44d2e2b4521d506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:27 UTC) from {username='slave'} at 127.0.0.1:33944
I20260927 21:44:27.725278 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 20795
I20260927 21:44:27.725229 21118 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e477e6f0b04a99a44d2e2b4521d506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:44:27 UTC) from {username='slave'} at 127.0.0.1:37036
I20260927 21:44:27.732829 21228 tablet_replica.cc:330] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: stopping tablet replica
I20260927 21:44:27.733693 21226 tablet_replica.cc:330] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: stopping tablet replica
I20260927 21:44:27.733741 21228 raft_consensus.cc:2229] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:27.734143 21226 raft_consensus.cc:2229] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:44:27.734326 21228 raft_consensus.cc:2258] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:27.734884 21226 raft_consensus.cc:2258] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:27.736644 21228 ts_tablet_manager.cc:1905] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:27.737352 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.726615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.0.202.129:35175)
0927 21:44:27.726844 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.726853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.727054 (+ 201us) server_negotiation.cc:407] Connection header received
0927 21:44:27.727317 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.727322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.727443 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.727525 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:27.728234 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.729425 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.731472 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.732226 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.732376 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.735563 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:27.735574 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:27.735590 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:27.736491 (+ 901us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:27.736567 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:27.736592 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:27.736677 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.736920 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.737023 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:44:27.738479 21226 ts_tablet_manager.cc:1905] T a4e477e6f0b04a99a44d2e2b4521d506 P a20fdac163eb47eabbe433a52d7b2645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:27.738601 20917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:27.740237 21227 tablet_replica.cc:330] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: stopping tablet replica
I20260927 21:44:27.740865 21227 raft_consensus.cc:2229] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:44:27.741520 21227 raft_consensus.cc:2258] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:44:27.743861 21227 ts_tablet_manager.cc:1905] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:44:27.746080 21228 ts_tablet_manager.cc:1918] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:27.746251 21228 log.cc:1196] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-2/wal/wals/a4e477e6f0b04a99a44d2e2b4521d506
I20260927 21:44:27.746773 21228 ts_tablet_manager.cc:1939] T a4e477e6f0b04a99a44d2e2b4521d506 P 8574c6ef52ea4a17a3ed24f39292c5a2: Deleting consensus metadata
I20260927 21:44:27.748755 20722 catalog_manager.cc:4841] TS 8574c6ef52ea4a17a3ed24f39292c5a2 (127.0.202.131:36247): tablet a4e477e6f0b04a99a44d2e2b4521d506 (table test-table [id=1b67be01d379482f99c04d0d986012d9]) successfully deleted
I20260927 21:44:27.750288 21227 ts_tablet_manager.cc:1918] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:44:27.750465 21227 log.cc:1196] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Deleting WAL directory at /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790545135802652-810-0/minicluster-data/ts-1/wal/wals/a4e477e6f0b04a99a44d2e2b4521d506
I20260927 21:44:27.750927 21227 ts_tablet_manager.cc:1939] T a4e477e6f0b04a99a44d2e2b4521d506 P 18b9253ad9b342e0aede019964b99490: Deleting consensus metadata
I20260927 21:44:27.752550 20723 catalog_manager.cc:4841] TS 18b9253ad9b342e0aede019964b99490 (127.0.202.130:33977): tablet a4e477e6f0b04a99a44d2e2b4521d506 (table test-table [id=1b67be01d379482f99c04d0d986012d9]) successfully deleted
I20260927 21:44:27.839493 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 20795
W20260927 21:44:27.858701 20722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.202.129:35175 (error 108)
I20260927 21:44:27.859459 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 20926
I20260927 21:44:27.860844 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.860263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53078)
0927 21:44:27.860487 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.860632 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 21:44:27.861158 20722 catalog_manager.cc:4584] TS a20fdac163eb47eabbe433a52d7b2645 (127.0.202.129:35175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4e477e6f0b04a99a44d2e2b4521d506: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
I20260927 21:44:27.866395 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.860528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.0.202.130:33977)
0927 21:44:27.860704 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.860710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.860729 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:44:27.860791 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.860795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.860866 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.860974 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:27.861384 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.862115 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.863374 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.863774 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.863902 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.865054 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:27.865063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:27.865076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:27.865642 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:27.865688 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:27.865707 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:27.865825 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.865963 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.866083 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:44:27.867064 21048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:27.915225 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.914576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53092)
0927 21:44:27.914902 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.915010 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260927 21:44:27.916265 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.915912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53094)
0927 21:44:27.916069 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.916136 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:44:27.959740 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 20926
I20260927 21:44:27.976791 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.976328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53104)
0927 21:44:27.976536 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.976648 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 21:44:27.977937 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.977567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53118)
0927 21:44:27.977722 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:27.977809 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 21:44:27.979123 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21056
I20260927 21:44:27.986435 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:27.979868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.0.202.131:36247)
0927 21:44:27.980079 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:27.980086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:27.980107 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:44:27.980369 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:27.980377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:27.980481 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:27.980601 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:27.981075 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.981864 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.983230 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:27.983675 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:27.983823 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:27.985048 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:27.985058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:27.985074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:27.985685 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:27.985737 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:27.985758 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:27.985892 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:27.986030 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:44:27.986136 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:44:27.987105 21178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:28.058859 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:28.058325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53126)
0927 21:44:28.058561 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:28.058661 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 21:44:28.060163 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:28.059720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.129:35175 (local address 127.0.0.1:53136)
0927 21:44:28.059947 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:44:28.060034 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.129:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 21:44:28.080401 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21056
I20260927 21:44:28.097761 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 20705
I20260927 21:44:28.098950 20766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:28.210237 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 20705
2026-09-27T21:44:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2901 ms)
[----------] 6 tests from EncryptionPolicyTest (22723 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:28 dist-test-slave-25dq krb5kdc[21237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:28 dist-test-slave-25dq krb5kdc[21237](info): set up 2 sockets
Sep 27 21:44:28 dist-test-slave-25dq krb5kdc[21237](info): commencing operation
krb5kdc: starting...
W20260927 21:44:29.345024 810 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:29 dist-test-slave-25dq krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:29.617244 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:29.965579 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:29.965984 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:29.966166 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:29.973690 21250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:29.973876 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:29.974012 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:29.974138 21250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:29.974232 21250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:29.984675 21250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:29.986155 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:29.987612 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:30.000375 21256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:30.000314 21255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:30.000568 21258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:30.001691 21250 server_base.cc:1034] running on GCE node
I20260927 21:44:30.002527 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:30.004031 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:30.005242 21250 hybrid_clock.cc:648] HybridClock initialized: now 1790545470005165 us; error 64 us; skew 500 ppm
I20260927 21:44:30.005653 21250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:30 dist-test-slave-25dq krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:30.009279 21250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:30.011101 21250 webserver.cc:460] Webserver started at http://127.0.0.1:43491/ using document root <none> and password file <none>
I20260927 21:44:30.011889 21250 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:30.012048 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:30.012457 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:30.015761 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "9eba8150df784afcaa7944473151e5af"
format_stamp: "Formatted at 2026-09-27 21:44:30 on dist-test-slave-25dq"
server_key: "0c5e251e35d144c49df5d4909b7b14ae"
server_key_iv: "bfabe374b3cdf60887f28ecc30deb0c2"
server_key_version: "encryptionkey@0"
I20260927 21:44:30.016709 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "9eba8150df784afcaa7944473151e5af"
format_stamp: "Formatted at 2026-09-27 21:44:30 on dist-test-slave-25dq"
server_key: "0c5e251e35d144c49df5d4909b7b14ae"
server_key_iv: "bfabe374b3cdf60887f28ecc30deb0c2"
server_key_version: "encryptionkey@0"
I20260927 21:44:30.022766 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:44:30.027032 21265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:30.028828 21250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:44:30.029032 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "9eba8150df784afcaa7944473151e5af"
format_stamp: "Formatted at 2026-09-27 21:44:30 on dist-test-slave-25dq"
server_key: "0c5e251e35d144c49df5d4909b7b14ae"
server_key_iv: "bfabe374b3cdf60887f28ecc30deb0c2"
server_key_version: "encryptionkey@0"
I20260927 21:44:30.029247 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:30.047726 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:30.052874 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:30.053236 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:30.073899 21250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37297
I20260927 21:44:30.073931 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37297 every 8 connection(s)
I20260927 21:44:30.075417 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:30.079687 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:30.084714 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21250
I20260927 21:44:30.084944 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:30.085637 810 external_mini_cluster.cc:1428] Setting key 26740f341ffb6eeeb7dffebab1513e84
I20260927 21:44:30.092746 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af: Bootstrap starting.
I20260927 21:44:30.096915 21317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:30.098192 21317 log.cc:824] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:30.101547 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af: No bootstrap required, opened a new log
Sep 27 21:44:30 dist-test-slave-25dq krb5kdc[21237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:44:30.113801 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:30.087682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40601 (local address 127.0.0.1:37297)
0927 21:44:30.088651 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:30.088669 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:30.088726 (+ 57us) server_negotiation.cc:407] Connection header received
0927 21:44:30.091311 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:30.091341 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:30.091894 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:30.092383 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:30.093832 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:30.096090 (+ 2258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:30.097519 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:30.098599 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:30.098820 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:30.103424 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:30.103438 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:30.103462 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:30.103545 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:30.108193 (+ 4648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:30.109177 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:30.109188 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:30.109197 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:30.109442 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:30.109994 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:30.110000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:30.110003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:30.110882 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:30.111258 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:30.111781 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 21:44:30.112187 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":404,"threads_started":1}
I20260927 21:44:30.113926 21317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eba8150df784afcaa7944473151e5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37297 } }
I20260927 21:44:30.114459 21317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:30.114598 21317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eba8150df784afcaa7944473151e5af, State: Initialized, Role: FOLLOWER
I20260927 21:44:30.115361 21317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eba8150df784afcaa7944473151e5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37297 } }
I20260927 21:44:30.115651 21317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:30.115844 21317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:30.116070 21317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:30.118848 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:30.119225 21317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eba8150df784afcaa7944473151e5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37297 } }
I20260927 21:44:30.119771 21317 leader_election.cc:302] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eba8150df784afcaa7944473151e5af; no voters:
I20260927 21:44:30.120512 21317 leader_election.cc:288] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:30.120657 21322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:30.123438 21317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:30.124948 21322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [term 1 LEADER]: Becoming Leader. State: Replica: 9eba8150df784afcaa7944473151e5af, State: Running, Role: LEADER
I20260927 21:44:30.125780 21322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eba8150df784afcaa7944473151e5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37297 } }
I20260927 21:44:30.129575 21324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eba8150df784afcaa7944473151e5af. Latest consensus state: current_term: 1 leader_uuid: "9eba8150df784afcaa7944473151e5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eba8150df784afcaa7944473151e5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37297 } } }
I20260927 21:44:30.130173 21324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:30.130899 21323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eba8150df784afcaa7944473151e5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eba8150df784afcaa7944473151e5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37297 } } }
I20260927 21:44:30.131249 21323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:30.138957 21330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:30.146067 21330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:30.158495 21330 catalog_manager.cc:1347] Generated new cluster ID: d731c0da77be43d69700376dedc7e746
I20260927 21:44:30.158627 21330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:30.169528 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:30.138089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40831 (local address 127.0.0.1:37297)
0927 21:44:30.138321 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:30.138331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:30.147856 (+ 9525us) server_negotiation.cc:407] Connection header received
0927 21:44:30.148092 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:30.148098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:30.148189 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:30.148347 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:30.150596 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:30.151816 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:30.153618 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:30.154486 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:30.154643 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:30.157924 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:30.157931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:30.157936 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:30.157972 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:30.163547 (+ 5575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:30.165417 (+ 1870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:30.165427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:30.165434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:30.165659 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:30.166535 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:30.166544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:30.166551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:30.167180 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:30.167546 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:30.168602 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0927 21:44:30.168903 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:44:30.176928 21330 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:30.179193 21330 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:30.188220 21330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eba8150df784afcaa7944473151e5af: Generated new TSK 0
I20260927 21:44:30.189456 21330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:30.203789 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21250
I20260927 21:44:30.205577 21312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:30.297843 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21250
W20260927 21:44:30.316145 21235 connection.cc:535] client connection to 127.0.0.1:37297 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37297): Connection reset by peer (error 104)
2026-09-27T21:44:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:30 dist-test-slave-25dq krb5kdc[21350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:30 dist-test-slave-25dq krb5kdc[21350](info): set up 2 sockets
Sep 27 21:44:30 dist-test-slave-25dq krb5kdc[21350](info): commencing operation
krb5kdc: starting...
W20260927 21:44:31.488910 810 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:31 dist-test-slave-25dq krb5kdc[21350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:31.760730 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:32.116802 21363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:32.117203 21363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:32.117383 21363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:32.125137 21363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:32.125334 21363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:32.125445 21363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:32.125573 21363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:32.125660 21363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:32.136475 21363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:32.137995 21363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:32.139475 21363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:32.152208 21371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:32.152328 21369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:32.152503 21368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:32.153342 21363 server_base.cc:1034] running on GCE node
I20260927 21:44:32.154155 21363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:32.155668 21363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:32.156924 21363 hybrid_clock.cc:648] HybridClock initialized: now 1790545472156891 us; error 53 us; skew 500 ppm
I20260927 21:44:32.157364 21363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:32 dist-test-slave-25dq krb5kdc[21350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:32.160840 21363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:32.162642 21363 webserver.cc:460] Webserver started at http://127.0.0.1:44883/ using document root <none> and password file <none>
I20260927 21:44:32.163376 21363 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:32.163527 21363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:32.163847 21363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:32.167178 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "e65755e921fe44d6ac190edecb860ecf"
format_stamp: "Formatted at 2026-09-27 21:44:32 on dist-test-slave-25dq"
server_key: "0c5d18133e651f02c94750d6355164fa"
server_key_iv: "0776bd6ba894af3c32a0eabac0242673"
server_key_version: "encryptionkey@0"
I20260927 21:44:32.168040 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "e65755e921fe44d6ac190edecb860ecf"
format_stamp: "Formatted at 2026-09-27 21:44:32 on dist-test-slave-25dq"
server_key: "0c5d18133e651f02c94750d6355164fa"
server_key_iv: "0776bd6ba894af3c32a0eabac0242673"
server_key_version: "encryptionkey@0"
I20260927 21:44:32.173833 21363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:44:32.178124 21378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:32.179812 21363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:44:32.180159 21363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "e65755e921fe44d6ac190edecb860ecf"
format_stamp: "Formatted at 2026-09-27 21:44:32 on dist-test-slave-25dq"
server_key: "0c5d18133e651f02c94750d6355164fa"
server_key_iv: "0776bd6ba894af3c32a0eabac0242673"
server_key_version: "encryptionkey@0"
I20260927 21:44:32.180459 21363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:32.193677 21363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:32.198412 21363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:32.198728 21363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:32.218734 21363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33859
I20260927 21:44:32.218788 21429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33859 every 8 connection(s)
I20260927 21:44:32.220198 21363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:32.224074 21430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:32.228093 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21363
I20260927 21:44:32.228317 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:32.228874 810 external_mini_cluster.cc:1428] Setting key 26773239144f3528e36d7afc1f7b4ed0
I20260927 21:44:32.237197 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf: Bootstrap starting.
I20260927 21:44:32.240881 21430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:32.242017 21430 log.cc:824] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:32.242614 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:32.230689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45293 (local address 127.0.0.1:33859)
0927 21:44:32.231408 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:32.231420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:32.231460 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:44:32.233206 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:32.233228 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:32.233729 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:32.234053 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:32.234559 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:32.235905 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:32.237760 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:32.238498 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:32.238647 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:32.240094 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:32.240109 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:32.240123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:32.240753 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:32.240830 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:32.240855 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:32.241043 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:32.241291 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 21:44:32.241553 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":331,"threads_started":1}
I20260927 21:44:32.244863 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf: No bootstrap required, opened a new log
I20260927 21:44:32.246559 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:32.255169 21430 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65755e921fe44d6ac190edecb860ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33859 } }
I20260927 21:44:32.255652 21430 raft_consensus.cc:374] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:32.255749 21430 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65755e921fe44d6ac190edecb860ecf, State: Initialized, Role: FOLLOWER
I20260927 21:44:32.256460 21430 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65755e921fe44d6ac190edecb860ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33859 } }
I20260927 21:44:32.256718 21430 raft_consensus.cc:388] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:32.256891 21430 raft_consensus.cc:482] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:32.257103 21430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:32.258976 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:32.251999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46055 (local address 127.0.0.1:33859)
0927 21:44:32.252231 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:32.252240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:32.252536 (+ 296us) server_negotiation.cc:407] Connection header received
0927 21:44:32.252752 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:32.252756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:32.252818 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:32.252900 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:32.253496 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:32.254349 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:32.256180 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:32.256816 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:32.256944 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:32.258150 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:32.258156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:32.258160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:32.258178 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:32.258217 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:32.258234 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:32.258327 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:32.258532 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 21:44:32.258631 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 21:44:32.259964 21430 raft_consensus.cc:504] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65755e921fe44d6ac190edecb860ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33859 } }
I20260927 21:44:32.260609 21430 leader_election.cc:302] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e65755e921fe44d6ac190edecb860ecf; no voters:
I20260927 21:44:32.261241 21430 leader_election.cc:288] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:32.261384 21440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:32.262137 21440 raft_consensus.cc:686] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [term 1 LEADER]: Becoming Leader. State: Replica: e65755e921fe44d6ac190edecb860ecf, State: Running, Role: LEADER
I20260927 21:44:32.262739 21440 consensus_queue.cc:237] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65755e921fe44d6ac190edecb860ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33859 } }
I20260927 21:44:32.263608 21430 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:32.266134 21442 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e65755e921fe44d6ac190edecb860ecf. Latest consensus state: current_term: 1 leader_uuid: "e65755e921fe44d6ac190edecb860ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65755e921fe44d6ac190edecb860ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33859 } } }
I20260927 21:44:32.266078 21441 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e65755e921fe44d6ac190edecb860ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65755e921fe44d6ac190edecb860ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33859 } } }
I20260927 21:44:32.267293 21441 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:32.266649 21442 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:32.269235 21449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:32.274083 21449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:32.286952 21449 catalog_manager.cc:1347] Generated new cluster ID: 357c0ed91bb648d5aed659d2ec95ce8b
I20260927 21:44:32.287093 21449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:32.306063 21449 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:32.307713 21449 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:32.314666 21449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e65755e921fe44d6ac190edecb860ecf: Generated new TSK 0
I20260927 21:44:32.315732 21449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:32.321847 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21363
W20260927 21:44:32.323043 21425 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45293
W20260927 21:44:32.323819 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:32.324018 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21363
W20260927 21:44:32.341473 21348 connection.cc:535] client connection to 127.0.0.1:33859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33859): Connection reset by peer (error 104)
2026-09-27T21:44:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:32 dist-test-slave-25dq krb5kdc[21461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:32 dist-test-slave-25dq krb5kdc[21461](info): set up 2 sockets
Sep 27 21:44:32 dist-test-slave-25dq krb5kdc[21461](info): commencing operation
krb5kdc: starting...
W20260927 21:44:33.504866 810 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.008s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:33 dist-test-slave-25dq krb5kdc[21461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:33.776615 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:34.126114 21474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:34.126503 21474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:34.126694 21474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:34.135013 21474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:34.135182 21474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:34.135300 21474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:34.135428 21474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:34.135515 21474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:34.146679 21474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:34.148092 21474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:34.149497 21474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:34.162032 21479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:34.162122 21480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:34.162168 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:34.165939 21474 server_base.cc:1034] running on GCE node
I20260927 21:44:34.166810 21474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:34.168453 21474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:34.169693 21474 hybrid_clock.cc:648] HybridClock initialized: now 1790545474169641 us; error 33 us; skew 500 ppm
I20260927 21:44:34.170095 21474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:34 dist-test-slave-25dq krb5kdc[21461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:34.180743 21474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:34.182633 21474 webserver.cc:460] Webserver started at http://127.0.0.1:35023/ using document root <none> and password file <none>
I20260927 21:44:34.183423 21474 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:34.183526 21474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:34.183900 21474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:34.187168 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "5faaf544408d4da585456e45110b654c"
format_stamp: "Formatted at 2026-09-27 21:44:34 on dist-test-slave-25dq"
server_key: "c5e79d41d6c776ad4540d5cc6ac80cec"
server_key_iv: "fff6d90589b5a8af6a99170249da58e7"
server_key_version: "encryptionkey@0"
I20260927 21:44:34.188010 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "5faaf544408d4da585456e45110b654c"
format_stamp: "Formatted at 2026-09-27 21:44:34 on dist-test-slave-25dq"
server_key: "c5e79d41d6c776ad4540d5cc6ac80cec"
server_key_iv: "fff6d90589b5a8af6a99170249da58e7"
server_key_version: "encryptionkey@0"
I20260927 21:44:34.193786 21474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:44:34.197531 21489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:34.199023 21474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:34.199235 21474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "5faaf544408d4da585456e45110b654c"
format_stamp: "Formatted at 2026-09-27 21:44:34 on dist-test-slave-25dq"
server_key: "c5e79d41d6c776ad4540d5cc6ac80cec"
server_key_iv: "fff6d90589b5a8af6a99170249da58e7"
server_key_version: "encryptionkey@0"
I20260927 21:44:34.199468 21474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:34.205821 21474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:34.210425 21474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:34.210740 21474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:34.230786 21474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41197
I20260927 21:44:34.230844 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41197 every 8 connection(s)
I20260927 21:44:34.232309 21474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:34.233698 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21474
I20260927 21:44:34.233879 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:34.234508 810 external_mini_cluster.cc:1428] Setting key efcdb76bfced5c876f6affe640e226c6
I20260927 21:44:34.237212 21541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:34.242895 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:34.236729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39465 (local address 127.0.0.1:41197)
0927 21:44:34.237576 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:34.237589 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:34.237631 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:44:34.239330 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:34.239351 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:34.239697 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:34.240032 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:34.241227 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:34.241234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:34.241250 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:34.241270 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:34.241341 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:34.241360 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:34.241533 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:34.241651 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:44:34.241860 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":382,"threads_started":1}
I20260927 21:44:34.246632 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:34.250000 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c: Bootstrap starting.
I20260927 21:44:34.254195 21541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:34.254674 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:34.252075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33211 (local address 127.0.0.1:41197)
0927 21:44:34.252338 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:34.252346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:34.252663 (+ 317us) server_negotiation.cc:407] Connection header received
0927 21:44:34.252798 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:34.252802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:34.252871 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:34.252988 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:34.253912 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:34.253918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:34.253925 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:34.253948 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:34.254007 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:34.254032 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:34.254169 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:34.254293 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 21:44:34.254351 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:44:34.255519 21541 log.cc:824] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:34.258383 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c: No bootstrap required, opened a new log
I20260927 21:44:34.265892 21541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faaf544408d4da585456e45110b654c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260927 21:44:34.266319 21541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:34.266426 21541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5faaf544408d4da585456e45110b654c, State: Initialized, Role: FOLLOWER
I20260927 21:44:34.267062 21541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faaf544408d4da585456e45110b654c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260927 21:44:34.267314 21541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:34.267488 21541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:34.267699 21541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:34.270109 21541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faaf544408d4da585456e45110b654c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260927 21:44:34.270653 21541 leader_election.cc:302] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5faaf544408d4da585456e45110b654c; no voters:
I20260927 21:44:34.271309 21541 leader_election.cc:288] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:34.271494 21551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:34.272629 21551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [term 1 LEADER]: Becoming Leader. State: Replica: 5faaf544408d4da585456e45110b654c, State: Running, Role: LEADER
I20260927 21:44:34.273478 21551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faaf544408d4da585456e45110b654c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } }
I20260927 21:44:34.273988 21541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:34.276863 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5faaf544408d4da585456e45110b654c. Latest consensus state: current_term: 1 leader_uuid: "5faaf544408d4da585456e45110b654c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faaf544408d4da585456e45110b654c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } } }
I20260927 21:44:34.276927 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5faaf544408d4da585456e45110b654c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faaf544408d4da585456e45110b654c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41197 } } }
I20260927 21:44:34.277262 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:34.277325 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:34.282788 21560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:34.287533 21560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:34.297366 21560 catalog_manager.cc:1347] Generated new cluster ID: 2af390f46f7b4a7595123d323ab324f8
I20260927 21:44:34.297501 21560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:34.321797 21560 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:34.322333 21560 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:34.328866 21560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5faaf544408d4da585456e45110b654c: Generated new TSK 0
I20260927 21:44:34.329756 21560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:34.342449 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21474
W20260927 21:44:34.343971 21536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39465
W20260927 21:44:34.344843 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:34.344954 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21474
2026-09-27T21:44:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2020 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:34 dist-test-slave-25dq krb5kdc[21572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:34 dist-test-slave-25dq krb5kdc[21572](info): set up 2 sockets
Sep 27 21:44:34 dist-test-slave-25dq krb5kdc[21572](info): commencing operation
krb5kdc: starting...
W20260927 21:44:35.484994 810 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:35 dist-test-slave-25dq krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:35.760265 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:36.097764 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:36.098361 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:36.098495 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:36.106539 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:36.106662 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:36.106709 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:36.106776 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:36.106834 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:36.117118 21585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:36.118408 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:36.119760 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:36.131887 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:36.132040 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:36.131970 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:36.133710 21585 server_base.cc:1034] running on GCE node
I20260927 21:44:36.134634 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:36.136219 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:36.137545 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790545476137482 us; error 52 us; skew 500 ppm
I20260927 21:44:36.138022 21585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:36 dist-test-slave-25dq krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:36.141613 21585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:36.143296 21585 webserver.cc:460] Webserver started at http://127.0.0.1:38993/ using document root <none> and password file <none>
I20260927 21:44:36.144071 21585 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:36.144223 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:36.144582 21585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:36.147796 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "fe041e4cade640ddbf598e4f59110fa6"
format_stamp: "Formatted at 2026-09-27 21:44:36 on dist-test-slave-25dq"
server_key: "a3b791d99741555d4a0211a94c673a16"
server_key_iv: "78cdd4b1515d7428abd741dad179953d"
server_key_version: "encryptionkey@0"
I20260927 21:44:36.148662 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "fe041e4cade640ddbf598e4f59110fa6"
format_stamp: "Formatted at 2026-09-27 21:44:36 on dist-test-slave-25dq"
server_key: "a3b791d99741555d4a0211a94c673a16"
server_key_iv: "78cdd4b1515d7428abd741dad179953d"
server_key_version: "encryptionkey@0"
I20260927 21:44:36.154184 21585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:44:36.157852 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:36.159567 21585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:36.159881 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "fe041e4cade640ddbf598e4f59110fa6"
format_stamp: "Formatted at 2026-09-27 21:44:36 on dist-test-slave-25dq"
server_key: "a3b791d99741555d4a0211a94c673a16"
server_key_iv: "78cdd4b1515d7428abd741dad179953d"
server_key_version: "encryptionkey@0"
I20260927 21:44:36.160148 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:36.196856 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:36.201905 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:36.202242 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:36.221613 21585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38317
I20260927 21:44:36.221663 21651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38317 every 8 connection(s)
I20260927 21:44:36.223131 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:36.227161 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:36.227584 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21585
I20260927 21:44:36.227772 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:36.228353 810 external_mini_cluster.cc:1428] Setting key 899dbbf3bd6b7f7760283b83664d103c
I20260927 21:44:36.241571 21652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6: Bootstrap starting.
Sep 27 21:44:36 dist-test-slave-25dq krb5kdc[21572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:44:36.245388 21652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:36.246757 21652 log.cc:824] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:36.250331 21652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6: No bootstrap required, opened a new log
I20260927 21:44:36.253741 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:36.230146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40661 (local address 127.0.0.1:38317)
0927 21:44:36.230961 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:36.230973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:36.231011 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:44:36.233489 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:36.233513 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:36.233924 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:36.234271 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:36.235979 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:36.237357 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:36.239161 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:36.239838 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:36.239979 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:36.245008 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:36.245020 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:36.245039 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:36.245088 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:36.248999 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:36.249817 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:36.249826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:36.249835 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:36.250041 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:36.250476 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:36.250481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:36.250484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:36.251216 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:36.251516 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:36.251899 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 21:44:36.252380 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":407,"threads_started":1}
I20260927 21:44:36.257638 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:36.259667 21652 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe041e4cade640ddbf598e4f59110fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260927 21:44:36.260048 21652 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:36.260174 21652 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe041e4cade640ddbf598e4f59110fa6, State: Initialized, Role: FOLLOWER
I20260927 21:44:36.260880 21652 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe041e4cade640ddbf598e4f59110fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260927 21:44:36.261171 21652 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:36.261318 21652 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:36.261484 21652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:36.264496 21652 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe041e4cade640ddbf598e4f59110fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260927 21:44:36.265091 21652 leader_election.cc:302] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe041e4cade640ddbf598e4f59110fa6; no voters:
I20260927 21:44:36.265779 21652 leader_election.cc:288] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:36.265873 21662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:36.267472 21662 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [term 1 LEADER]: Becoming Leader. State: Replica: fe041e4cade640ddbf598e4f59110fa6, State: Running, Role: LEADER
I20260927 21:44:36.268111 21662 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe041e4cade640ddbf598e4f59110fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } }
I20260927 21:44:36.268318 21652 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:36.276991 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:36.262149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49205 (local address 127.0.0.1:38317)
0927 21:44:36.262352 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:36.262360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:36.262743 (+ 383us) server_negotiation.cc:407] Connection header received
0927 21:44:36.262939 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:36.262947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:36.263021 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:36.263141 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:36.264652 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:36.265491 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:36.267251 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:36.267968 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:36.268120 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:36.270912 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:36.270917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:36.270921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:36.270952 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:36.274153 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:36.274912 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:36.274919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:36.274923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:36.275055 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:36.275421 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:36.275426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:36.275430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:36.275742 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:36.276000 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:36.276354 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:44:36.276592 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:44:36.277842 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe041e4cade640ddbf598e4f59110fa6. Latest consensus state: current_term: 1 leader_uuid: "fe041e4cade640ddbf598e4f59110fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe041e4cade640ddbf598e4f59110fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } } }
I20260927 21:44:36.278219 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:36.279206 21663 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe041e4cade640ddbf598e4f59110fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe041e4cade640ddbf598e4f59110fa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38317 } } }
I20260927 21:44:36.279542 21663 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:36.281826 21670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:36.287484 21670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:36.303622 21670 catalog_manager.cc:1347] Generated new cluster ID: 6e5173e978a34d8aba87cb80c1e1e14e
I20260927 21:44:36.303776 21670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:36.319360 21670 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:36.321045 21670 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:36.330420 21670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe041e4cade640ddbf598e4f59110fa6: Generated new TSK 0
I20260927 21:44:36.331485 21670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:36.337385 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21585
I20260927 21:44:36.338691 21647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:36.432912 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21585
2026-09-27T21:44:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:36 dist-test-slave-25dq krb5kdc[21684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:36 dist-test-slave-25dq krb5kdc[21684](info): set up 2 sockets
Sep 27 21:44:36 dist-test-slave-25dq krb5kdc[21684](info): commencing operation
krb5kdc: starting...
W20260927 21:44:37.572847 810 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:37 dist-test-slave-25dq krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:37.845306 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:38.183955 21697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:38.184353 21697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:38.184536 21697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:38.192109 21697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:38.192241 21697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:38.192353 21697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:38.192466 21697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:38.192535 21697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:38.203045 21697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:38.204548 21697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:38.205940 21697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:38.218876 21702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:38.218935 21703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:38.218935 21705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:38.220709 21697 server_base.cc:1034] running on GCE node
I20260927 21:44:38.221585 21697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:38.223141 21697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:38.224422 21697 hybrid_clock.cc:648] HybridClock initialized: now 1790545478224376 us; error 43 us; skew 500 ppm
I20260927 21:44:38.224821 21697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:38 dist-test-slave-25dq krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:38.228560 21697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:38.230350 21697 webserver.cc:460] Webserver started at http://127.0.0.1:41133/ using document root <none> and password file <none>
I20260927 21:44:38.231120 21697 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:38.231279 21697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:38.231627 21697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:38.234964 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "018c211086c743dab117c01663f3cc94"
format_stamp: "Formatted at 2026-09-27 21:44:38 on dist-test-slave-25dq"
server_key: "c3debb0f88b207f159cdab2fc676188e"
server_key_iv: "7fa9943adadc4823eb451327f62bc4a0"
server_key_version: "encryptionkey@0"
I20260927 21:44:38.235813 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "018c211086c743dab117c01663f3cc94"
format_stamp: "Formatted at 2026-09-27 21:44:38 on dist-test-slave-25dq"
server_key: "c3debb0f88b207f159cdab2fc676188e"
server_key_iv: "7fa9943adadc4823eb451327f62bc4a0"
server_key_version: "encryptionkey@0"
I20260927 21:44:38.241603 21697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:44:38.245285 21712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:38.246877 21697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:38.247110 21697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "018c211086c743dab117c01663f3cc94"
format_stamp: "Formatted at 2026-09-27 21:44:38 on dist-test-slave-25dq"
server_key: "c3debb0f88b207f159cdab2fc676188e"
server_key_iv: "7fa9943adadc4823eb451327f62bc4a0"
server_key_version: "encryptionkey@0"
I20260927 21:44:38.247329 21697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:38.276644 21697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:38.281662 21697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:38.282004 21697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:38.302708 21697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36841
I20260927 21:44:38.302769 21763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36841 every 8 connection(s)
I20260927 21:44:38.304175 21697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:38.308357 21764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:38.312528 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21697
I20260927 21:44:38.312728 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:38.313244 810 external_mini_cluster.cc:1428] Setting key e9f49125a2982ddb73e78105ec5c32a4
I20260927 21:44:38.321628 21764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94: Bootstrap starting.
I20260927 21:44:38.325083 21764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:38.326285 21764 log.cc:824] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:38.327381 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:38.315286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43665 (local address 127.0.0.1:36841)
0927 21:44:38.316123 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:38.316135 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:38.316176 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:44:38.317901 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:38.317933 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:38.318407 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:38.318871 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:38.319470 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:38.320928 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:38.322334 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:38.323040 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:38.323263 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:38.324631 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:38.324640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:38.324655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:38.325352 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:38.325448 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:38.325476 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:38.325627 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:38.325887 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:44:38.326194 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":386,"threads_started":1}
I20260927 21:44:38.329298 21764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94: No bootstrap required, opened a new log
I20260927 21:44:38.331395 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:38.337844 21764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018c211086c743dab117c01663f3cc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36841 } }
I20260927 21:44:38.338244 21764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:38.338351 21764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 018c211086c743dab117c01663f3cc94, State: Initialized, Role: FOLLOWER
I20260927 21:44:38.338994 21764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018c211086c743dab117c01663f3cc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36841 } }
I20260927 21:44:38.339250 21764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:38.339392 21764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:38.339569 21764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:38.342272 21764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018c211086c743dab117c01663f3cc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36841 } }
I20260927 21:44:38.342880 21764 leader_election.cc:302] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 018c211086c743dab117c01663f3cc94; no voters:
I20260927 21:44:38.343142 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:38.336031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38243 (local address 127.0.0.1:36841)
0927 21:44:38.336259 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:38.336269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:38.336659 (+ 390us) server_negotiation.cc:407] Connection header received
0927 21:44:38.336880 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:38.336884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:38.336951 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:38.337075 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:38.337627 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:38.338542 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:38.340161 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:38.340818 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:38.340973 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:38.342213 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:38.342218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:38.342222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:38.342241 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:38.342285 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:38.342302 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:38.342409 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:38.342678 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 21:44:38.342773 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:44:38.343636 21764 leader_election.cc:288] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:38.343789 21774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:38.345379 21774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [term 1 LEADER]: Becoming Leader. State: Replica: 018c211086c743dab117c01663f3cc94, State: Running, Role: LEADER
I20260927 21:44:38.345937 21774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018c211086c743dab117c01663f3cc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36841 } }
I20260927 21:44:38.346148 21764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:38.350057 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "018c211086c743dab117c01663f3cc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018c211086c743dab117c01663f3cc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36841 } } }
I20260927 21:44:38.350009 21776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 018c211086c743dab117c01663f3cc94. Latest consensus state: current_term: 1 leader_uuid: "018c211086c743dab117c01663f3cc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018c211086c743dab117c01663f3cc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36841 } } }
I20260927 21:44:38.350553 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:38.350553 21776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:38.351725 21782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:38.360772 21782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:38.370237 21782 catalog_manager.cc:1347] Generated new cluster ID: 3f3216b0cca64cd793a5fc1bafe5a0cb
I20260927 21:44:38.370373 21782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:38.405903 21782 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:38.407883 21782 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:38.430847 21782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 018c211086c743dab117c01663f3cc94: Generated new TSK 0
I20260927 21:44:38.431960 21782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:38.441556 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21697
W20260927 21:44:38.442721 21759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43665
W20260927 21:44:38.443434 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:38.443588 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21697
2026-09-27T21:44:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:38 dist-test-slave-25dq krb5kdc[21795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:38 dist-test-slave-25dq krb5kdc[21795](info): set up 2 sockets
Sep 27 21:44:38 dist-test-slave-25dq krb5kdc[21795](info): commencing operation
krb5kdc: starting...
W20260927 21:44:39.568862 810 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:39 dist-test-slave-25dq krb5kdc[21795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:39.845106 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:40.190089 21808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:40.190826 21808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:40.191002 21808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:40.198547 21808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:40.198735 21808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:40.198850 21808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:40.198966 21808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:40.199065 21808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:40.209266 21808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:40.210765 21808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:40.212208 21808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:40.225140 21813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:40.225180 21814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:40.225107 21816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:40.225860 21808 server_base.cc:1034] running on GCE node
I20260927 21:44:40.226648 21808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:40.228092 21808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:40.229307 21808 hybrid_clock.cc:648] HybridClock initialized: now 1790545480229250 us; error 76 us; skew 500 ppm
I20260927 21:44:40.229653 21808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:40 dist-test-slave-25dq krb5kdc[21795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:40.237936 21808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:40.239733 21808 webserver.cc:460] Webserver started at http://127.0.0.1:42937/ using document root <none> and password file <none>
I20260927 21:44:40.240537 21808 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:40.240700 21808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:40.241042 21808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:40.244096 21808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "7dbe7d45959e4c3c9a1578688d4ab534"
format_stamp: "Formatted at 2026-09-27 21:44:40 on dist-test-slave-25dq"
server_key: "28409bcdb47717ec81d1a26b041e21c7"
server_key_iv: "3b52046f0fd30e4714cd334e24c82711"
server_key_version: "encryptionkey@0"
I20260927 21:44:40.244938 21808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "7dbe7d45959e4c3c9a1578688d4ab534"
format_stamp: "Formatted at 2026-09-27 21:44:40 on dist-test-slave-25dq"
server_key: "28409bcdb47717ec81d1a26b041e21c7"
server_key_iv: "3b52046f0fd30e4714cd334e24c82711"
server_key_version: "encryptionkey@0"
I20260927 21:44:40.250232 21808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:44:40.253881 21823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:40.255584 21808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:40.255784 21808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "7dbe7d45959e4c3c9a1578688d4ab534"
format_stamp: "Formatted at 2026-09-27 21:44:40 on dist-test-slave-25dq"
server_key: "28409bcdb47717ec81d1a26b041e21c7"
server_key_iv: "3b52046f0fd30e4714cd334e24c82711"
server_key_version: "encryptionkey@0"
I20260927 21:44:40.256038 21808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:40.262307 21808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:40.266721 21808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:40.267045 21808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:40.286630 21808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43995
I20260927 21:44:40.286686 21874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43995 every 8 connection(s)
I20260927 21:44:40.288122 21808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:40.292019 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21808
I20260927 21:44:40.292244 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:40.292337 21875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:40.292821 810 external_mini_cluster.cc:1428] Setting key 026ab1e79e5d3dc6abfb88412e340bed
I20260927 21:44:40.301111 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:40.294679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42187 (local address 127.0.0.1:43995)
0927 21:44:40.295482 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:40.295497 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:40.295543 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:44:40.297439 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:40.297463 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:40.297837 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:40.298286 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:40.299375 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:40.299381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:40.299395 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:40.299409 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:40.299477 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:40.299497 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:40.299749 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:40.299906 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 21:44:40.300095 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":339,"threads_started":1}
I20260927 21:44:40.305127 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:40.306336 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534: Bootstrap starting.
I20260927 21:44:40.309906 21875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:40.311122 21875 log.cc:824] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:40.312561 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:40.309815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45899 (local address 127.0.0.1:43995)
0927 21:44:40.310085 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:40.310093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:40.310453 (+ 360us) server_negotiation.cc:407] Connection header received
0927 21:44:40.310653 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:40.310658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:40.310716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:40.310841 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:40.311847 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:40.311852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:40.311857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:40.311877 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:40.311940 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:40.311958 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:40.312099 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:40.312187 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 21:44:40.312253 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 21:44:40.314464 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534: No bootstrap required, opened a new log
I20260927 21:44:40.322355 21875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260927 21:44:40.322818 21875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:40.322927 21875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbe7d45959e4c3c9a1578688d4ab534, State: Initialized, Role: FOLLOWER
I20260927 21:44:40.323545 21875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260927 21:44:40.323825 21875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:40.324028 21875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:40.324241 21875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:40.326519 21875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260927 21:44:40.327029 21875 leader_election.cc:302] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dbe7d45959e4c3c9a1578688d4ab534; no voters:
I20260927 21:44:40.327633 21875 leader_election.cc:288] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:40.327796 21885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:40.328521 21885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [term 1 LEADER]: Becoming Leader. State: Replica: 7dbe7d45959e4c3c9a1578688d4ab534, State: Running, Role: LEADER
I20260927 21:44:40.329017 21885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } }
I20260927 21:44:40.330070 21875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:40.332063 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dbe7d45959e4c3c9a1578688d4ab534. Latest consensus state: current_term: 1 leader_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } } }
I20260927 21:44:40.332050 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe7d45959e4c3c9a1578688d4ab534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43995 } } }
I20260927 21:44:40.332681 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:40.332851 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:40.337396 21891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:40.343024 21891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:40.352202 21891 catalog_manager.cc:1347] Generated new cluster ID: 7d141da3c25440838bb48f0a9f5186c2
I20260927 21:44:40.352419 21891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:40.373189 21891 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:40.373768 21891 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:40.399587 21891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dbe7d45959e4c3c9a1578688d4ab534: Generated new TSK 0
I20260927 21:44:40.400738 21891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:40.420871 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21808
W20260927 21:44:40.422641 21870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42187
W20260927 21:44:40.423456 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21808: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:40.423631 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21808
2026-09-27T21:44:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:40 dist-test-slave-25dq krb5kdc[21906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:40 dist-test-slave-25dq krb5kdc[21906](info): set up 2 sockets
Sep 27 21:44:40 dist-test-slave-25dq krb5kdc[21906](info): commencing operation
krb5kdc: starting...
W20260927 21:44:41.568907 810 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:41 dist-test-slave-25dq krb5kdc[21906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:41.844883 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:42.188217 21919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:42.188643 21919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:42.188817 21919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:42.196420 21919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:42.196606 21919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:42.196715 21919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:42.196834 21919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:42.196913 21919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:42.207265 21919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:42.208746 21919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:42.210143 21919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:42.222899 21924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:42.222990 21927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:42.223179 21925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:42.223778 21919 server_base.cc:1034] running on GCE node
I20260927 21:44:42.224686 21919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:42.226217 21919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:42.227543 21919 hybrid_clock.cc:648] HybridClock initialized: now 1790545482227481 us; error 43 us; skew 500 ppm
I20260927 21:44:42.227946 21919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:42 dist-test-slave-25dq krb5kdc[21906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:42.238412 21919 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:42.240268 21919 webserver.cc:460] Webserver started at http://127.0.0.1:33009/ using document root <none> and password file <none>
I20260927 21:44:42.241350 21919 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:42.241516 21919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:42.241865 21919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:42.245236 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "6869408a06cf4ee88691dcaf21cd42b5"
format_stamp: "Formatted at 2026-09-27 21:44:42 on dist-test-slave-25dq"
server_key: "d2f20d97c2bded1097c520461c6566ee"
server_key_iv: "39001434a154120ddf184e3b5727840d"
server_key_version: "encryptionkey@0"
I20260927 21:44:42.246044 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "6869408a06cf4ee88691dcaf21cd42b5"
format_stamp: "Formatted at 2026-09-27 21:44:42 on dist-test-slave-25dq"
server_key: "d2f20d97c2bded1097c520461c6566ee"
server_key_iv: "39001434a154120ddf184e3b5727840d"
server_key_version: "encryptionkey@0"
I20260927 21:44:42.251552 21919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:44:42.255280 21934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:42.256923 21919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:44:42.257148 21919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "6869408a06cf4ee88691dcaf21cd42b5"
format_stamp: "Formatted at 2026-09-27 21:44:42 on dist-test-slave-25dq"
server_key: "d2f20d97c2bded1097c520461c6566ee"
server_key_iv: "39001434a154120ddf184e3b5727840d"
server_key_version: "encryptionkey@0"
I20260927 21:44:42.257377 21919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:42.285276 21919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:42.290143 21919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:42.290469 21919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:42.310647 21919 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44021
I20260927 21:44:42.310676 21985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44021 every 8 connection(s)
I20260927 21:44:42.312158 21919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:42.312748 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 21919
I20260927 21:44:42.313040 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:42.313692 810 external_mini_cluster.cc:1428] Setting key f8d827bde897c73abdef0a6c364f4cc4
I20260927 21:44:42.316952 21986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:42.327840 21986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5: Bootstrap starting.
Sep 27 21:44:42 dist-test-slave-25dq krb5kdc[21906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:44:42.331118 21986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:42.332204 21986 log.cc:824] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:42.335318 21986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5: No bootstrap required, opened a new log
I20260927 21:44:42.338967 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:42.315685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:53395 (local address 127.0.0.1:44021)
0927 21:44:42.316558 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:42.316571 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:42.316617 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:44:42.318658 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:42.318682 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:42.319124 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:42.319595 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:42.321221 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:42.322697 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:42.324092 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:42.324920 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:42.325176 (+ 256us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:42.330133 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:42.330142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:42.330159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:42.330205 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:42.334176 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:42.335099 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:42.335106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:42.335112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:42.335290 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:42.335856 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:42.335861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:42.335863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:42.336522 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:42.336941 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:42.337517 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 21:44:42.337854 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":323,"threads_started":1}
I20260927 21:44:42.343428 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:42.344583 21986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6869408a06cf4ee88691dcaf21cd42b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260927 21:44:42.344983 21986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:42.345108 21986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6869408a06cf4ee88691dcaf21cd42b5, State: Initialized, Role: FOLLOWER
I20260927 21:44:42.345733 21986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6869408a06cf4ee88691dcaf21cd42b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260927 21:44:42.345984 21986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:42.346138 21986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:42.346344 21986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:42.349386 21986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6869408a06cf4ee88691dcaf21cd42b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260927 21:44:42.350107 21986 leader_election.cc:302] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6869408a06cf4ee88691dcaf21cd42b5; no voters:
I20260927 21:44:42.350759 21986 leader_election.cc:288] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:42.350975 21996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:42.353224 21986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:42.356979 21996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [term 1 LEADER]: Becoming Leader. State: Replica: 6869408a06cf4ee88691dcaf21cd42b5, State: Running, Role: LEADER
I20260927 21:44:42.357818 21996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6869408a06cf4ee88691dcaf21cd42b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } }
I20260927 21:44:42.361313 21997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6869408a06cf4ee88691dcaf21cd42b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6869408a06cf4ee88691dcaf21cd42b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } } }
I20260927 21:44:42.361816 21997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:42.362455 21998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6869408a06cf4ee88691dcaf21cd42b5. Latest consensus state: current_term: 1 leader_uuid: "6869408a06cf4ee88691dcaf21cd42b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6869408a06cf4ee88691dcaf21cd42b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44021 } } }
I20260927 21:44:42.362835 21998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:42.363408 22003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:42.371403 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:42.348366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:42439 (local address 127.0.0.1:44021)
0927 21:44:42.348552 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:42.348560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:42.348992 (+ 432us) server_negotiation.cc:407] Connection header received
0927 21:44:42.349211 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:42.349217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:42.349334 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:42.349447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:42.352081 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:42.353339 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:42.355633 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:42.356917 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:42.357883 (+ 966us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:42.360758 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:42.360763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:42.360768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:42.360844 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:42.365279 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:42.366205 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:42.366213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:42.366218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:42.366480 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:42.367230 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:42.367240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:42.367250 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:42.367837 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:42.368210 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:42.369963 (+ 1753us) server_negotiation.cc:299] Negotiation successful
0927 21:44:42.370384 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:44:42.374452 22003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:42.385492 22003 catalog_manager.cc:1347] Generated new cluster ID: 30cf53847e5d459ab6fdc81c3296911e
I20260927 21:44:42.385625 22003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:42.407404 22003 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:42.409385 22003 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:42.420892 22003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6869408a06cf4ee88691dcaf21cd42b5: Generated new TSK 0
I20260927 21:44:42.422052 22003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:42.431658 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 21919
I20260927 21:44:42.433131 21981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:42.553754 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 21919
W20260927 21:44:42.571694 21904 connection.cc:535] client connection to 127.0.0.1:44021 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44021): Connection reset by peer (error 104)
2026-09-27T21:44:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2130 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:42 dist-test-slave-25dq krb5kdc[22018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:42 dist-test-slave-25dq krb5kdc[22018](info): set up 2 sockets
Sep 27 21:44:42 dist-test-slave-25dq krb5kdc[22018](info): commencing operation
krb5kdc: starting...
W20260927 21:44:43.680915 810 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:43 dist-test-slave-25dq krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:43.973052 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:44.330140 22031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:44.330677 22031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:44.330983 22031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:44.339159 22031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:44.339274 22031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:44.339351 22031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:44.339454 22031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:44.339542 22031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:44.349905 22031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:44.351270 22031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:44.352692 22031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:44.364949 22036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:44.364992 22039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:44.364959 22037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:44.365804 22031 server_base.cc:1034] running on GCE node
I20260927 21:44:44.366640 22031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:44.368122 22031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:44.369388 22031 hybrid_clock.cc:648] HybridClock initialized: now 1790545484369295 us; error 73 us; skew 500 ppm
I20260927 21:44:44.369812 22031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:44 dist-test-slave-25dq krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:44.373371 22031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:44.374902 22031 webserver.cc:460] Webserver started at http://127.0.0.1:44633/ using document root <none> and password file <none>
I20260927 21:44:44.375571 22031 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:44.375675 22031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:44.375936 22031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:44.379132 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "ee2048cd3ae642659ac6acd2978ac4aa"
format_stamp: "Formatted at 2026-09-27 21:44:44 on dist-test-slave-25dq"
server_key: "ce0a05d40a4bcc2b36aabe1a5c8ce30b"
server_key_iv: "66babb02b51beff1f0ab43ccaf6b996a"
server_key_version: "encryptionkey@0"
I20260927 21:44:44.380055 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "ee2048cd3ae642659ac6acd2978ac4aa"
format_stamp: "Formatted at 2026-09-27 21:44:44 on dist-test-slave-25dq"
server_key: "ce0a05d40a4bcc2b36aabe1a5c8ce30b"
server_key_iv: "66babb02b51beff1f0ab43ccaf6b996a"
server_key_version: "encryptionkey@0"
I20260927 21:44:44.385926 22031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:44:44.389921 22046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:44.391536 22031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:44.391753 22031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ee2048cd3ae642659ac6acd2978ac4aa"
format_stamp: "Formatted at 2026-09-27 21:44:44 on dist-test-slave-25dq"
server_key: "ce0a05d40a4bcc2b36aabe1a5c8ce30b"
server_key_iv: "66babb02b51beff1f0ab43ccaf6b996a"
server_key_version: "encryptionkey@0"
I20260927 21:44:44.391990 22031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:44.417487 22031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:44.422940 22031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:44.423301 22031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:44.445036 22031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38207
I20260927 21:44:44.445080 22097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38207 every 8 connection(s)
I20260927 21:44:44.446681 22031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:44.450809 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:44.451538 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22031
I20260927 21:44:44.451773 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:44.452369 810 external_mini_cluster.cc:1428] Setting key e4202ffe2061e6011c80943076a6c921
I20260927 21:44:44.464999 22098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa: Bootstrap starting.
I20260927 21:44:44.466815 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:44.454431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:38033 (local address 127.0.0.1:38207)
0927 21:44:44.455259 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:44.455271 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:44.455315 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:44:44.457057 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:44.457083 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:44.457506 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:44.457851 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:44.458464 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:44.459761 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:44.461744 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:44.462621 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:44.462784 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:44.464132 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:44.464140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:44.464161 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:44.464927 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:44.465013 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:44.465038 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:44.465242 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:44.465504 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:44:44.465714 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":382,"threads_started":1}
I20260927 21:44:44.468390 22098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:44.469625 22098 log.cc:824] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:44.470872 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:44.473168 22098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa: No bootstrap required, opened a new log
I20260927 21:44:44.484406 22098 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260927 21:44:44.485041 22098 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:44.485208 22098 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2048cd3ae642659ac6acd2978ac4aa, State: Initialized, Role: FOLLOWER
I20260927 21:44:44.485275 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:44.476259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:46633 (local address 127.0.0.1:38207)
0927 21:44:44.476543 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:44.476553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:44.478194 (+ 1641us) server_negotiation.cc:407] Connection header received
0927 21:44:44.478392 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:44.478397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:44.478472 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:44.478564 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:44.479233 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:44.480176 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:44.482201 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:44.482725 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:44.482840 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:44.484241 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:44.484246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:44.484249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:44.484269 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:44.484405 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:44.484425 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:44.484522 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:44.484746 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 21:44:44.484794 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:44:44.486022 22098 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260927 21:44:44.486315 22098 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:44.486496 22098 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:44.486707 22098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:44.489573 22098 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260927 21:44:44.490121 22098 leader_election.cc:302] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee2048cd3ae642659ac6acd2978ac4aa; no voters:
I20260927 21:44:44.490701 22098 leader_election.cc:288] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:44.490863 22108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:44.491675 22108 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [term 1 LEADER]: Becoming Leader. State: Replica: ee2048cd3ae642659ac6acd2978ac4aa, State: Running, Role: LEADER
I20260927 21:44:44.492281 22108 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260927 21:44:44.493145 22098 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:44.495530 22109 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } } }
I20260927 21:44:44.495883 22109 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:44.495420 22110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee2048cd3ae642659ac6acd2978ac4aa. Latest consensus state: current_term: 1 leader_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2048cd3ae642659ac6acd2978ac4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } } }
I20260927 21:44:44.497344 22110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:44.498101 22116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:44.505048 22116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:44.514995 22116 catalog_manager.cc:1347] Generated new cluster ID: 7b100635da13409e816fc9fee923bf40
I20260927 21:44:44.515153 22116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:44.524665 22116 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:44.526005 22116 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:44.533214 22116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee2048cd3ae642659ac6acd2978ac4aa: Generated new TSK 0
I20260927 21:44:44.534119 22116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:44.548847 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22031
W20260927 21:44:44.550251 22093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:38033
W20260927 21:44:44.551414 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22031: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:44.551578 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22031
W20260927 21:44:44.568764 22016 connection.cc:535] client connection to 127.0.0.1:38207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38207): Connection reset by peer (error 104)
2026-09-27T21:44:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:44 dist-test-slave-25dq krb5kdc[22129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:44 dist-test-slave-25dq krb5kdc[22129](info): set up 2 sockets
Sep 27 21:44:44 dist-test-slave-25dq krb5kdc[22129](info): commencing operation
krb5kdc: starting...
W20260927 21:44:45.685453 810 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:45 dist-test-slave-25dq krb5kdc[22129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:45.964234 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:46.306612 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:46.306963 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:46.307133 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:46.314553 22142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:46.314725 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:46.314837 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:46.314949 22142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:46.315033 22142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:46.325219 22142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:46.326583 22142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:46.327924 22142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:46.340338 22147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:46.340406 22148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:46.340541 22150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:46.341128 22142 server_base.cc:1034] running on GCE node
I20260927 21:44:46.341888 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:46.343232 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:46.344445 22142 hybrid_clock.cc:648] HybridClock initialized: now 1790545486344382 us; error 60 us; skew 500 ppm
I20260927 21:44:46.344818 22142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:46 dist-test-slave-25dq krb5kdc[22129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:46.351893 22142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:46.353624 22142 webserver.cc:460] Webserver started at http://127.0.0.1:38361/ using document root <none> and password file <none>
I20260927 21:44:46.354365 22142 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:46.354514 22142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:46.354887 22142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:46.358057 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "c8856df5196146c6995b8b2c7d5b707d"
format_stamp: "Formatted at 2026-09-27 21:44:46 on dist-test-slave-25dq"
server_key: "ef12487269941af5e50aa9d79cb1aa60"
server_key_iv: "35df2ceb5e3a63710f764583f4128a11"
server_key_version: "encryptionkey@0"
I20260927 21:44:46.358870 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "c8856df5196146c6995b8b2c7d5b707d"
format_stamp: "Formatted at 2026-09-27 21:44:46 on dist-test-slave-25dq"
server_key: "ef12487269941af5e50aa9d79cb1aa60"
server_key_iv: "35df2ceb5e3a63710f764583f4128a11"
server_key_version: "encryptionkey@0"
I20260927 21:44:46.364653 22142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 21:44:46.368463 22157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:46.370102 22142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:46.370323 22142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "c8856df5196146c6995b8b2c7d5b707d"
format_stamp: "Formatted at 2026-09-27 21:44:46 on dist-test-slave-25dq"
server_key: "ef12487269941af5e50aa9d79cb1aa60"
server_key_iv: "35df2ceb5e3a63710f764583f4128a11"
server_key_version: "encryptionkey@0"
I20260927 21:44:46.370554 22142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:46.376726 22142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:46.381655 22142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:46.382014 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:46.402442 22142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35213
I20260927 21:44:46.402484 22208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35213 every 8 connection(s)
I20260927 21:44:46.403975 22142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:46.407891 22209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:46.410748 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22142
I20260927 21:44:46.410934 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:46.411466 810 external_mini_cluster.cc:1428] Setting key c538625843be30dfcf2083fdb69b804a
I20260927 21:44:46.420265 22209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d: Bootstrap starting.
I20260927 21:44:46.420274 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:46.413374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:57707 (local address 127.0.0.1:35213)
0927 21:44:46.414280 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:46.414293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:46.414336 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:44:46.416044 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:46.416067 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:46.416496 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:46.416853 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:46.417962 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:46.417970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:46.417984 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:46.418004 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:46.418089 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:46.418115 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:46.418374 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:46.418501 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:44:46.419228 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":466,"threads_started":1}
I20260927 21:44:46.423666 22209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:46.425096 22209 log.cc:824] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:46.425151 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:46.428282 22209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d: No bootstrap required, opened a new log
I20260927 21:44:46.433460 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:46.430485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:39637 (local address 127.0.0.1:35213)
0927 21:44:46.430689 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:46.430696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:46.431239 (+ 543us) server_negotiation.cc:407] Connection header received
0927 21:44:46.431446 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:46.431452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:46.431521 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:46.431638 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:46.432626 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:46.432632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:46.432637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:46.432660 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:46.432713 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:46.432735 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:46.432861 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:46.433023 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:44:46.433088 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:44:46.437069 22209 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8856df5196146c6995b8b2c7d5b707d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260927 21:44:46.437441 22209 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:46.437572 22209 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8856df5196146c6995b8b2c7d5b707d, State: Initialized, Role: FOLLOWER
I20260927 21:44:46.438215 22209 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8856df5196146c6995b8b2c7d5b707d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260927 21:44:46.438462 22209 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:46.438611 22209 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:46.438779 22209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:46.441426 22209 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8856df5196146c6995b8b2c7d5b707d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260927 21:44:46.441941 22209 leader_election.cc:302] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8856df5196146c6995b8b2c7d5b707d; no voters:
I20260927 21:44:46.442561 22209 leader_election.cc:288] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:46.442739 22219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:46.443418 22219 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [term 1 LEADER]: Becoming Leader. State: Replica: c8856df5196146c6995b8b2c7d5b707d, State: Running, Role: LEADER
I20260927 21:44:46.444010 22219 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8856df5196146c6995b8b2c7d5b707d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260927 21:44:46.444737 22209 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:46.446995 22220 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8856df5196146c6995b8b2c7d5b707d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8856df5196146c6995b8b2c7d5b707d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } } }
I20260927 21:44:46.448009 22220 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:46.448310 22221 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8856df5196146c6995b8b2c7d5b707d. Latest consensus state: current_term: 1 leader_uuid: "c8856df5196146c6995b8b2c7d5b707d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8856df5196146c6995b8b2c7d5b707d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } } }
I20260927 21:44:46.448750 22221 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:46.449929 22226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:46.455348 22226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:46.464470 22226 catalog_manager.cc:1347] Generated new cluster ID: 32905984c1d845d6abd09ed866d28e57
I20260927 21:44:46.464608 22226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:46.477458 22226 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:46.478052 22226 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:46.490710 22226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8856df5196146c6995b8b2c7d5b707d: Generated new TSK 0
I20260927 21:44:46.491989 22226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:46.504786 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22142
W20260927 21:44:46.506187 22204 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:57707
W20260927 21:44:46.507061 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22142: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:46.507220 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22142
2026-09-27T21:44:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:46 dist-test-slave-25dq krb5kdc[22240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:46 dist-test-slave-25dq krb5kdc[22240](info): set up 2 sockets
Sep 27 21:44:46 dist-test-slave-25dq krb5kdc[22240](info): commencing operation
krb5kdc: starting...
W20260927 21:44:47.668910 810 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:47 dist-test-slave-25dq krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:47.941615 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:48.285722 22253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:48.286123 22253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:48.286244 22253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:48.293803 22253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:48.293972 22253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:48.294023 22253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:48.294077 22253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:48.294116 22253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:48.304209 22253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:48.305591 22253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:48.307468 22253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:48.319883 22258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:48.319911 22261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:48.319880 22259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:48.320946 22253 server_base.cc:1034] running on GCE node
I20260927 21:44:48.321764 22253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:48.323264 22253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:48.324509 22253 hybrid_clock.cc:648] HybridClock initialized: now 1790545488324480 us; error 73 us; skew 500 ppm
I20260927 21:44:48.325006 22253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:48 dist-test-slave-25dq krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:48.328766 22253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:48.330749 22253 webserver.cc:460] Webserver started at http://127.0.0.1:35961/ using document root <none> and password file <none>
I20260927 21:44:48.331502 22253 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:48.331664 22253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:48.332026 22253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:48.335292 22253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "d851ae34df9b469788cc5328545aaac6"
format_stamp: "Formatted at 2026-09-27 21:44:48 on dist-test-slave-25dq"
server_key: "2c8b71f6ca494519f9fc2cc35cb4c2c7"
server_key_iv: "480f5ab4c081a08bfdb4f9ddc1a42ef5"
server_key_version: "encryptionkey@0"
I20260927 21:44:48.336134 22253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "d851ae34df9b469788cc5328545aaac6"
format_stamp: "Formatted at 2026-09-27 21:44:48 on dist-test-slave-25dq"
server_key: "2c8b71f6ca494519f9fc2cc35cb4c2c7"
server_key_iv: "480f5ab4c081a08bfdb4f9ddc1a42ef5"
server_key_version: "encryptionkey@0"
I20260927 21:44:48.341938 22253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:44:48.345738 22268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:48.347175 22253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:48.347373 22253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "d851ae34df9b469788cc5328545aaac6"
format_stamp: "Formatted at 2026-09-27 21:44:48 on dist-test-slave-25dq"
server_key: "2c8b71f6ca494519f9fc2cc35cb4c2c7"
server_key_iv: "480f5ab4c081a08bfdb4f9ddc1a42ef5"
server_key_version: "encryptionkey@0"
I20260927 21:44:48.347579 22253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:48.375756 22253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:48.380719 22253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:48.381034 22253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:48.400841 22253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41619
I20260927 21:44:48.400889 22319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41619 every 8 connection(s)
I20260927 21:44:48.402290 22253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:48.406414 22320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:48.408725 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22253
I20260927 21:44:48.408910 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:48.409363 810 external_mini_cluster.cc:1428] Setting key 06a15bdce0636f33d3d606e9769ee8ed
I20260927 21:44:48.418208 22320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6: Bootstrap starting.
I20260927 21:44:48.421519 22320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:48.422581 22320 log.cc:824] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:44:48 dist-test-slave-25dq krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:44:48.425940 22320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6: No bootstrap required, opened a new log
I20260927 21:44:48.432263 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:48.411372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:50773 (local address 127.0.0.1:41619)
0927 21:44:48.412169 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:48.412191 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:48.412234 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:44:48.414016 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:48.414039 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:48.414450 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:48.414844 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:48.416330 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:48.417636 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:48.418966 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:48.419643 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:48.419864 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:48.424352 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:48.424362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:48.424377 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:48.424416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:48.427678 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:48.428655 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:48.428662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:48.428669 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:48.428839 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:48.429324 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:48.429327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:48.429329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:48.429869 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:48.430105 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:48.430750 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 21:44:48.431206 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":351,"threads_started":1}
I20260927 21:44:48.435032 22320 raft_consensus.cc:348] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d851ae34df9b469788cc5328545aaac6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20260927 21:44:48.435480 22320 raft_consensus.cc:374] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:48.435664 22320 raft_consensus.cc:729] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d851ae34df9b469788cc5328545aaac6, State: Initialized, Role: FOLLOWER
I20260927 21:44:48.436437 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:48.436380 22320 consensus_queue.cc:260] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d851ae34df9b469788cc5328545aaac6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20260927 21:44:48.436661 22320 raft_consensus.cc:388] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:48.436781 22320 raft_consensus.cc:482] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:48.436930 22320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:48.440229 22320 raft_consensus.cc:504] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d851ae34df9b469788cc5328545aaac6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20260927 21:44:48.440900 22320 leader_election.cc:302] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d851ae34df9b469788cc5328545aaac6; no voters:
I20260927 21:44:48.441596 22320 leader_election.cc:288] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:48.441743 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:48.443192 22329 raft_consensus.cc:686] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [term 1 LEADER]: Becoming Leader. State: Replica: d851ae34df9b469788cc5328545aaac6, State: Running, Role: LEADER
I20260927 21:44:48.443977 22329 consensus_queue.cc:237] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d851ae34df9b469788cc5328545aaac6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20260927 21:44:48.444449 22320 sys_catalog.cc:564] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:48.446749 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d851ae34df9b469788cc5328545aaac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d851ae34df9b469788cc5328545aaac6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } } }
I20260927 21:44:48.447139 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:48.449023 22332 sys_catalog.cc:455] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d851ae34df9b469788cc5328545aaac6. Latest consensus state: current_term: 1 leader_uuid: "d851ae34df9b469788cc5328545aaac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d851ae34df9b469788cc5328545aaac6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } } }
I20260927 21:44:48.450078 22332 sys_catalog.cc:458] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:48.457672 22335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:48.466951 22335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:48.466993 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:48.441973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:39769 (local address 127.0.0.1:41619)
0927 21:44:48.442242 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:48.442253 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:48.443989 (+ 1736us) server_negotiation.cc:407] Connection header received
0927 21:44:48.444193 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:48.444198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:48.444285 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:48.444416 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:48.446623 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:48.448354 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:48.451047 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:48.452053 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:48.452194 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:48.457164 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:48.457176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:48.457184 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:48.457252 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:48.462233 (+ 4981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:48.463390 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:48.463396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:48.463400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:48.463553 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:48.464187 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:48.464192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:48.464196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:48.464631 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:48.465115 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:48.466346 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0927 21:44:48.466542 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 21:44:48.479056 22335 catalog_manager.cc:1347] Generated new cluster ID: 000e34be7b2c4ba2a03715fb48bcf7a8
I20260927 21:44:48.479215 22335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:48.511384 22335 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:48.513146 22335 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:48.522279 22335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d851ae34df9b469788cc5328545aaac6: Generated new TSK 0
I20260927 21:44:48.523241 22335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:48.533075 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22253
I20260927 21:44:48.534931 22315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:48.637074 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22253
W20260927 21:44:48.657285 22238 connection.cc:535] client connection to 127.0.0.1:41619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41619): Connection reset by peer (error 104)
2026-09-27T21:44:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2133 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:48 dist-test-slave-25dq krb5kdc[22353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:48 dist-test-slave-25dq krb5kdc[22353](info): set up 2 sockets
Sep 27 21:44:48 dist-test-slave-25dq krb5kdc[22353](info): commencing operation
krb5kdc: starting...
W20260927 21:44:49.781018 810 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:49 dist-test-slave-25dq krb5kdc[22353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:50.065002 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:50.405018 22366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:50.405393 22366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:50.405570 22366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:50.413014 22366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:50.413169 22366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:50.413285 22366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:50.413388 22366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:50.413472 22366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:50.423537 22366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:50.425084 22366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:50.426425 22366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:50.438365 22372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:50.438408 22374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:50.438585 22371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:50.439419 22366 server_base.cc:1034] running on GCE node
I20260927 21:44:50.440264 22366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:50.441793 22366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:50.443038 22366 hybrid_clock.cc:648] HybridClock initialized: now 1790545490442994 us; error 58 us; skew 500 ppm
I20260927 21:44:50.443410 22366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:50 dist-test-slave-25dq krb5kdc[22353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:50.446842 22366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:50.448524 22366 webserver.cc:460] Webserver started at http://127.0.0.1:39459/ using document root <none> and password file <none>
I20260927 21:44:50.449209 22366 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:50.449309 22366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:50.449586 22366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:50.452667 22366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "7c2c0ddf58d4473f9809f1909a251cb3"
format_stamp: "Formatted at 2026-09-27 21:44:50 on dist-test-slave-25dq"
server_key: "aa6cffd514ba55f43751cf2e3b0094f7"
server_key_iv: "425f0f984dfa3f1b9a2f29f8cf10cd6c"
server_key_version: "encryptionkey@0"
I20260927 21:44:50.453413 22366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "7c2c0ddf58d4473f9809f1909a251cb3"
format_stamp: "Formatted at 2026-09-27 21:44:50 on dist-test-slave-25dq"
server_key: "aa6cffd514ba55f43751cf2e3b0094f7"
server_key_iv: "425f0f984dfa3f1b9a2f29f8cf10cd6c"
server_key_version: "encryptionkey@0"
I20260927 21:44:50.459040 22366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 21:44:50.462805 22381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:50.464246 22366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:44:50.464498 22366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "7c2c0ddf58d4473f9809f1909a251cb3"
format_stamp: "Formatted at 2026-09-27 21:44:50 on dist-test-slave-25dq"
server_key: "aa6cffd514ba55f43751cf2e3b0094f7"
server_key_iv: "425f0f984dfa3f1b9a2f29f8cf10cd6c"
server_key_version: "encryptionkey@0"
I20260927 21:44:50.464680 22366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:50.481437 22366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:50.485855 22366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:50.486105 22366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:50.505921 22366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35055
I20260927 21:44:50.505975 22432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35055 every 8 connection(s)
I20260927 21:44:50.507488 22366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:50.511329 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22366
I20260927 21:44:50.511520 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:50.511813 22433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:50.512122 810 external_mini_cluster.cc:1428] Setting key 8046d5ff3e907fde1d7be504112abedd
I20260927 21:44:50.526510 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:50.514040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:51627 (local address 127.0.0.1:35055)
0927 21:44:50.514866 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:50.514877 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:50.514921 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:44:50.516802 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:50.516824 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:50.517245 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:50.517584 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:50.518266 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:50.519597 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:50.521732 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:50.522454 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:50.522587 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:50.524040 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:50.524048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:50.524066 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:50.524697 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:50.524782 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:50.524809 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:50.524946 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:50.525264 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 21:44:50.525485 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":395,"threads_started":1}
I20260927 21:44:50.527631 22433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3: Bootstrap starting.
I20260927 21:44:50.530467 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:50.531490 22433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:50.532938 22433 log.cc:824] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:50.535881 22433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3: No bootstrap required, opened a new log
I20260927 21:44:50.543919 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:50.536505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:50913 (local address 127.0.0.1:35055)
0927 21:44:50.536686 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:50.536694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:50.537125 (+ 431us) server_negotiation.cc:407] Connection header received
0927 21:44:50.537292 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:50.537297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:50.537379 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:50.537497 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:50.538057 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:50.539261 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:50.540806 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:50.541563 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:50.541704 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:50.543034 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:50.543042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:50.543047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:50.543070 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:50.543117 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:50.543136 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:50.543247 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:50.543483 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:44:50.543547 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:44:50.546047 22433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35055 } }
I20260927 21:44:50.546454 22433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:50.546573 22433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2c0ddf58d4473f9809f1909a251cb3, State: Initialized, Role: FOLLOWER
I20260927 21:44:50.547164 22433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35055 } }
I20260927 21:44:50.547430 22433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:50.547600 22433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:50.547811 22433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:50.550118 22433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35055 } }
I20260927 21:44:50.550630 22433 leader_election.cc:302] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c2c0ddf58d4473f9809f1909a251cb3; no voters:
I20260927 21:44:50.551223 22433 leader_election.cc:288] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:50.551414 22443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:50.552241 22443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 7c2c0ddf58d4473f9809f1909a251cb3, State: Running, Role: LEADER
I20260927 21:44:50.553018 22443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35055 } }
I20260927 21:44:50.553649 22433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:50.556205 22445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c2c0ddf58d4473f9809f1909a251cb3. Latest consensus state: current_term: 1 leader_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35055 } } }
I20260927 21:44:50.556828 22445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:50.556311 22444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2c0ddf58d4473f9809f1909a251cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35055 } } }
I20260927 21:44:50.557020 22444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:50.558480 22449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:50.563647 22449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:50.573243 22449 catalog_manager.cc:1347] Generated new cluster ID: 2ffd5e4fb4004ba8b8d0c24c76782c3f
I20260927 21:44:50.573370 22449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:50.602341 22449 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:50.604177 22449 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:50.619462 22449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c2c0ddf58d4473f9809f1909a251cb3: Generated new TSK 0
I20260927 21:44:50.620481 22449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:50.635953 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22366
W20260927 21:44:50.637215 22428 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:51627
W20260927 21:44:50.638513 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22366: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:50.638669 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22366
W20260927 21:44:50.657308 22351 connection.cc:535] client connection to 127.0.0.1:35055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35055): Connection reset by peer (error 104)
2026-09-27T21:44:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:50 dist-test-slave-25dq krb5kdc[22464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:50 dist-test-slave-25dq krb5kdc[22464](info): set up 2 sockets
Sep 27 21:44:50 dist-test-slave-25dq krb5kdc[22464](info): commencing operation
krb5kdc: starting...
W20260927 21:44:51.777037 810 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:51 dist-test-slave-25dq krb5kdc[22464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:44:52.044269 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:52.394197 22477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:52.394500 22477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:52.394593 22477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:52.401724 22477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:52.401847 22477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:52.401894 22477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:52.401958 22477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:52.402001 22477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:52.411823 22477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:52.413151 22477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:52.414403 22477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:52.426167 22483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:52.426142 22485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:52.426244 22482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:52.427043 22477 server_base.cc:1034] running on GCE node
I20260927 21:44:52.427783 22477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:52.429224 22477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:52.430444 22477 hybrid_clock.cc:648] HybridClock initialized: now 1790545492430400 us; error 26 us; skew 500 ppm
I20260927 21:44:52.430817 22477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:52 dist-test-slave-25dq krb5kdc[22464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:52.442003 22477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:44:52.443645 22477 webserver.cc:460] Webserver started at http://127.0.0.1:40949/ using document root <none> and password file <none>
I20260927 21:44:52.444442 22477 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:52.444588 22477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:52.444893 22477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:52.447849 22477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "6f38c57402fa410297a59611139d1bca"
format_stamp: "Formatted at 2026-09-27 21:44:52 on dist-test-slave-25dq"
server_key: "bb06087988e4cf30818dc87ffc1a4961"
server_key_iv: "9f5702f5e28256894ca08701e685b37c"
server_key_version: "encryptionkey@0"
I20260927 21:44:52.448661 22477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "6f38c57402fa410297a59611139d1bca"
format_stamp: "Formatted at 2026-09-27 21:44:52 on dist-test-slave-25dq"
server_key: "bb06087988e4cf30818dc87ffc1a4961"
server_key_iv: "9f5702f5e28256894ca08701e685b37c"
server_key_version: "encryptionkey@0"
I20260927 21:44:52.454037 22477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:44:52.458155 22492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:52.459565 22477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:44:52.459779 22477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "6f38c57402fa410297a59611139d1bca"
format_stamp: "Formatted at 2026-09-27 21:44:52 on dist-test-slave-25dq"
server_key: "bb06087988e4cf30818dc87ffc1a4961"
server_key_iv: "9f5702f5e28256894ca08701e685b37c"
server_key_version: "encryptionkey@0"
I20260927 21:44:52.459996 22477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:52.466167 22477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:52.470618 22477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:52.470934 22477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:52.490254 22477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45065
I20260927 21:44:52.490289 22543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45065 every 8 connection(s)
I20260927 21:44:52.491906 22477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:52.492635 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22477
I20260927 21:44:52.492837 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:52.493431 810 external_mini_cluster.cc:1428] Setting key 912c2253a2cee51aaba7e255d630634b
I20260927 21:44:52.496608 22544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:52.501329 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:52.495305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:60843 (local address 127.0.0.1:45065)
0927 21:44:52.496071 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:52.496083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:52.496126 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:44:52.497867 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:52.497890 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:52.498226 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:52.498609 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:52.499683 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:52.499688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:52.499701 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:52.499720 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:52.499781 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:52.499799 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:52.500040 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:52.500186 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:44:52.500421 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":314,"threads_started":1}
I20260927 21:44:52.505511 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:52.510203 22544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca: Bootstrap starting.
I20260927 21:44:52.514372 22544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:52.516474 22544 log.cc:824] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:52.518627 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:52.514991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:47445 (local address 127.0.0.1:45065)
0927 21:44:52.515345 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:52.515354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:52.515770 (+ 416us) server_negotiation.cc:407] Connection header received
0927 21:44:52.515989 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:52.515997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:52.516223 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:52.516416 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:52.517821 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:52.517826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:52.517832 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:52.517854 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:52.517914 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:52.517936 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:52.518067 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:52.518248 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 21:44:52.518327 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 21:44:52.520805 22544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca: No bootstrap required, opened a new log
I20260927 21:44:52.531111 22544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f38c57402fa410297a59611139d1bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260927 21:44:52.531634 22544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:52.531746 22544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f38c57402fa410297a59611139d1bca, State: Initialized, Role: FOLLOWER
I20260927 21:44:52.532502 22544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f38c57402fa410297a59611139d1bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260927 21:44:52.532799 22544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:52.532961 22544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:52.533160 22544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:52.536417 22544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f38c57402fa410297a59611139d1bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260927 21:44:52.537163 22544 leader_election.cc:302] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f38c57402fa410297a59611139d1bca; no voters:
I20260927 21:44:52.537921 22544 leader_election.cc:288] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:52.538009 22554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:52.538770 22554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [term 1 LEADER]: Becoming Leader. State: Replica: 6f38c57402fa410297a59611139d1bca, State: Running, Role: LEADER
I20260927 21:44:52.539332 22554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f38c57402fa410297a59611139d1bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260927 21:44:52.540350 22544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:52.543154 22555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f38c57402fa410297a59611139d1bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f38c57402fa410297a59611139d1bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } } }
I20260927 21:44:52.543697 22555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:52.545219 22556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f38c57402fa410297a59611139d1bca. Latest consensus state: current_term: 1 leader_uuid: "6f38c57402fa410297a59611139d1bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f38c57402fa410297a59611139d1bca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } } }
I20260927 21:44:52.546103 22556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:52.549006 22560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:52.557488 22560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:52.572930 22560 catalog_manager.cc:1347] Generated new cluster ID: 970b52e66e5040438b5f87759055075f
I20260927 21:44:52.573092 22560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:52.598691 22560 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:52.599247 22560 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:52.608057 22560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f38c57402fa410297a59611139d1bca: Generated new TSK 0
I20260927 21:44:52.609055 22560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:52.616971 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22477
W20260927 21:44:52.618455 22539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:60843
W20260927 21:44:52.619148 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:52.619300 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22477
2026-09-27T21:44:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:52 dist-test-slave-25dq krb5kdc[22575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:52 dist-test-slave-25dq krb5kdc[22575](info): set up 2 sockets
Sep 27 21:44:52 dist-test-slave-25dq krb5kdc[22575](info): commencing operation
krb5kdc: starting...
W20260927 21:44:53.776916 810 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:53 dist-test-slave-25dq krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:44:54.057116 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:37671
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:54.394984 22588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:54.395355 22588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:54.395532 22588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:54.403081 22588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:54.403237 22588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:54.403344 22588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:54.403463 22588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:54.403541 22588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:54.414060 22588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:37671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:54.415493 22588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:54.416925 22588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:54.429865 22593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:54.429804 22596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:54.430048 22594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:54.430706 22588 server_base.cc:1034] running on GCE node
I20260927 21:44:54.431471 22588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:54.433161 22588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:54.434371 22588 hybrid_clock.cc:648] HybridClock initialized: now 1790545494434307 us; error 65 us; skew 500 ppm
I20260927 21:44:54.434687 22588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:54 dist-test-slave-25dq krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:54.437934 22588 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:44:54.439481 22588 webserver.cc:460] Webserver started at http://127.0.202.190:42969/ using document root <none> and password file <none>
I20260927 21:44:54.440161 22588 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:54.440255 22588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:54.440600 22588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:54.443588 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "7c9c763ebf5544c385d709cc679dec3b"
format_stamp: "Formatted at 2026-09-27 21:44:54 on dist-test-slave-25dq"
server_key: "dcdaba682e82956330d5b72c1ec613ab"
server_key_iv: "77756180c154e1c54d06019df8a1e63a"
server_key_version: "encryptionkey@0"
I20260927 21:44:54.444335 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "7c9c763ebf5544c385d709cc679dec3b"
format_stamp: "Formatted at 2026-09-27 21:44:54 on dist-test-slave-25dq"
server_key: "dcdaba682e82956330d5b72c1ec613ab"
server_key_iv: "77756180c154e1c54d06019df8a1e63a"
server_key_version: "encryptionkey@0"
I20260927 21:44:54.449543 22588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:44:54.453346 22603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:54.454782 22588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:54.454975 22588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "7c9c763ebf5544c385d709cc679dec3b"
format_stamp: "Formatted at 2026-09-27 21:44:54 on dist-test-slave-25dq"
server_key: "dcdaba682e82956330d5b72c1ec613ab"
server_key_iv: "77756180c154e1c54d06019df8a1e63a"
server_key_version: "encryptionkey@0"
I20260927 21:44:54.455154 22588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:54.476758 22588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:54.481293 22588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:54.481539 22588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:54.501648 22588 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:37671
I20260927 21:44:54.501670 22654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:37671 every 8 connection(s)
I20260927 21:44:54.503154 22588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:54.507158 22655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:54.513568 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22588
I20260927 21:44:54.513803 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:54.514333 810 external_mini_cluster.cc:1428] Setting key f6f0904204a8bf491aff9d0634ec3981
I20260927 21:44:54.518628 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b: Bootstrap starting.
I20260927 21:44:54.521955 22655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:54.523088 22655 log.cc:824] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:54.526203 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b: No bootstrap required, opened a new log
Sep 27 21:44:54 dist-test-slave-25dq krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:44:54.535022 22655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9c763ebf5544c385d709cc679dec3b" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37671 } }
I20260927 21:44:54.535440 22655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:54.535584 22655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9c763ebf5544c385d709cc679dec3b, State: Initialized, Role: FOLLOWER
I20260927 21:44:54.536252 22655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9c763ebf5544c385d709cc679dec3b" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37671 } }
I20260927 21:44:54.536618 22655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:54.536784 22655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:54.536995 22655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:54.538687 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:54.516549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42157 (local address 127.0.202.190:37671)
0927 21:44:54.517386 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:54.517399 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:54.517442 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:44:54.519294 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:54.519318 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:54.519753 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:54.520197 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:54.521735 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:54.523209 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:54.524661 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:54.525364 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:54.525565 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:54.530160 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:54.530168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:54.530187 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:54.530231 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:54.533883 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:54.534808 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:54.534816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:54.534823 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:54.534996 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:54.535536 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:54.535544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:54.535547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:54.536369 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:54.536729 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:54.537213 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 21:44:54.537559 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":379,"threads_started":1}
I20260927 21:44:54.539635 22655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9c763ebf5544c385d709cc679dec3b" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37671 } }
I20260927 21:44:54.540197 22655 leader_election.cc:302] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c9c763ebf5544c385d709cc679dec3b; no voters:
I20260927 21:44:54.540930 22655 leader_election.cc:288] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:54.541080 22660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:54.542420 22660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [term 1 LEADER]: Becoming Leader. State: Replica: 7c9c763ebf5544c385d709cc679dec3b, State: Running, Role: LEADER
I20260927 21:44:54.543061 22660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9c763ebf5544c385d709cc679dec3b" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37671 } }
I20260927 21:44:54.543326 22655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:54.543406 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:54.555109 22661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c9c763ebf5544c385d709cc679dec3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9c763ebf5544c385d709cc679dec3b" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37671 } } }
I20260927 21:44:54.555626 22661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:54.556279 22662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c9c763ebf5544c385d709cc679dec3b. Latest consensus state: current_term: 1 leader_uuid: "7c9c763ebf5544c385d709cc679dec3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9c763ebf5544c385d709cc679dec3b" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 37671 } } }
I20260927 21:44:54.557503 22662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:54.566888 22674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:54.572997 22674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:54.582518 22674 catalog_manager.cc:1347] Generated new cluster ID: 29053b0ba24249888f3691b1f1588586
I20260927 21:44:54.582633 22674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:54.587984 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:54.568878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56599 (local address 127.0.202.190:37671)
0927 21:44:54.569096 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:54.569106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:54.569757 (+ 651us) server_negotiation.cc:407] Connection header received
0927 21:44:54.569940 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:54.569946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:54.570037 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:54.570174 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:54.572147 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:54.573346 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:54.575207 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:54.575909 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:54.576017 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:54.579568 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:54.579576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:54.579580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:44:54.579642 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:54.583577 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:54.584717 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:54.584724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:54.584729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:54.584873 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:44:54.585458 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:44:54.585463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:44:54.585466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:44:54.585918 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:54.586189 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:54.587410 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0927 21:44:54.587594 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:44:54.599157 22674 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:54.600616 22674 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:54.609063 22674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c9c763ebf5544c385d709cc679dec3b: Generated new TSK 0
I20260927 21:44:54.610219 22674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:54.621016 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22588
I20260927 21:44:54.622251 22650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:44:54.724797 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22588
W20260927 21:44:54.742714 22573 connection.cc:535] client connection to 127.0.202.190:37671 recv error: Network error: failed to read from TLS socket (remote: 127.0.202.190:37671): Connection reset by peer (error 104)
2026-09-27T21:44:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2104 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:54 dist-test-slave-25dq krb5kdc[22688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:54 dist-test-slave-25dq krb5kdc[22688](info): set up 2 sockets
Sep 27 21:44:54 dist-test-slave-25dq krb5kdc[22688](info): commencing operation
krb5kdc: starting...
W20260927 21:44:55.873648 810 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:56 dist-test-slave-25dq krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:44:56.148715 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41473
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:41473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:56.493409 22701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:56.493778 22701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:56.493955 22701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:56.502714 22701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:56.502874 22701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:56.502954 22701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:56.503072 22701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:56.503140 22701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:56.513831 22701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:41473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:56.515319 22701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:56.516782 22701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:56.530323 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:56.530571 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:56.530642 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:56.531857 22701 server_base.cc:1034] running on GCE node
I20260927 21:44:56.532718 22701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:56.534292 22701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:56.535560 22701 hybrid_clock.cc:648] HybridClock initialized: now 1790545496535506 us; error 44 us; skew 500 ppm
I20260927 21:44:56.535995 22701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:56 dist-test-slave-25dq krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:56.539899 22701 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:44:56.541917 22701 webserver.cc:460] Webserver started at http://127.0.202.190:36353/ using document root <none> and password file <none>
I20260927 21:44:56.542654 22701 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:56.542770 22701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:56.543046 22701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:56.546530 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "2c94ca23d4cc4b739b556b086dff2d72"
format_stamp: "Formatted at 2026-09-27 21:44:56 on dist-test-slave-25dq"
server_key: "81ec380ee9244dead329e4a216e74a8b"
server_key_iv: "ddb846fc7df35a152692ab0292cb6615"
server_key_version: "encryptionkey@0"
I20260927 21:44:56.547358 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "2c94ca23d4cc4b739b556b086dff2d72"
format_stamp: "Formatted at 2026-09-27 21:44:56 on dist-test-slave-25dq"
server_key: "81ec380ee9244dead329e4a216e74a8b"
server_key_iv: "ddb846fc7df35a152692ab0292cb6615"
server_key_version: "encryptionkey@0"
I20260927 21:44:56.553228 22701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:44:56.557298 22716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:56.558946 22701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:44:56.559142 22701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "2c94ca23d4cc4b739b556b086dff2d72"
format_stamp: "Formatted at 2026-09-27 21:44:56 on dist-test-slave-25dq"
server_key: "81ec380ee9244dead329e4a216e74a8b"
server_key_iv: "ddb846fc7df35a152692ab0292cb6615"
server_key_version: "encryptionkey@0"
I20260927 21:44:56.559341 22701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:56.573379 22701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:56.578207 22701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:56.578508 22701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:56.597440 22701 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:41473
I20260927 21:44:56.597462 22767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:41473 every 8 connection(s)
I20260927 21:44:56.598855 22701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:56.602607 22768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:56.605345 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22701
I20260927 21:44:56.605515 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:56.605971 810 external_mini_cluster.cc:1428] Setting key abc61224c30e67c0f903ce883ccd60a1
I20260927 21:44:56.616039 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72: Bootstrap starting.
I20260927 21:44:56.620182 22768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:56.621522 22768 log.cc:824] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:56.621672 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:56.607919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45725 (local address 127.0.202.190:41473)
0927 21:44:56.608710 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:56.608729 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:56.608781 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:44:56.611117 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:56.611146 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:56.611695 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:56.612147 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:56.612715 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:56.614773 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:56.616351 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:56.617338 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:56.617512 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:56.618810 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:56.618820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:56.618838 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:56.619567 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:56.619656 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:56.619692 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:56.619858 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:56.620130 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:44:56.620345 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":341,"threads_started":1}
I20260927 21:44:56.625052 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72: No bootstrap required, opened a new log
I20260927 21:44:56.626379 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:56.638772 22768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94ca23d4cc4b739b556b086dff2d72" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41473 } }
I20260927 21:44:56.639430 22768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:56.639613 22768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c94ca23d4cc4b739b556b086dff2d72, State: Initialized, Role: FOLLOWER
I20260927 21:44:56.640539 22768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94ca23d4cc4b739b556b086dff2d72" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41473 } }
I20260927 21:44:56.640973 22768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:56.641211 22768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:56.641418 22768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:56.644584 22768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94ca23d4cc4b739b556b086dff2d72" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41473 } }
I20260927 21:44:56.645147 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:56.636997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39567 (local address 127.0.202.190:41473)
0927 21:44:56.638597 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:56.638606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:56.638628 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:44:56.638730 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:56.638735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:56.638836 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:56.638960 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:56.639660 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:56.640963 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:56.642438 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:44:56.642938 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:44:56.643084 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:44:56.644332 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:56.644339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:56.644343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:56.644367 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:56.644449 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:56.644467 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:56.644604 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:56.644810 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 21:44:56.644867 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409}
I20260927 21:44:56.645409 22768 leader_election.cc:302] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c94ca23d4cc4b739b556b086dff2d72; no voters:
I20260927 21:44:56.646224 22768 leader_election.cc:288] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:56.646347 22778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:56.647571 22778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [term 1 LEADER]: Becoming Leader. State: Replica: 2c94ca23d4cc4b739b556b086dff2d72, State: Running, Role: LEADER
I20260927 21:44:56.648138 22778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94ca23d4cc4b739b556b086dff2d72" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41473 } }
I20260927 21:44:56.648975 22768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:56.651574 22780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c94ca23d4cc4b739b556b086dff2d72. Latest consensus state: current_term: 1 leader_uuid: "2c94ca23d4cc4b739b556b086dff2d72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94ca23d4cc4b739b556b086dff2d72" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41473 } } }
I20260927 21:44:56.651561 22779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c94ca23d4cc4b739b556b086dff2d72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c94ca23d4cc4b739b556b086dff2d72" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41473 } } }
I20260927 21:44:56.652204 22780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:56.652204 22779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:56.653648 22784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:56.661161 22784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:56.671914 22784 catalog_manager.cc:1347] Generated new cluster ID: 258e82beb1e544fdbeb79a12509a48c3
I20260927 21:44:56.672058 22784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:56.718116 22784 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:56.719993 22784 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:56.737497 22784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c94ca23d4cc4b739b556b086dff2d72: Generated new TSK 0
I20260927 21:44:56.738457 22784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:56.752151 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22701
W20260927 21:44:56.753319 22763 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45725
W20260927 21:44:56.754091 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22701: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:56.754244 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22701
W20260927 21:44:56.771646 22686 connection.cc:535] client connection to 127.0.202.190:41473 recv error: Network error: failed to read from TLS socket (remote: 127.0.202.190:41473): Connection reset by peer (error 104)
2026-09-27T21:44:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:56 dist-test-slave-25dq krb5kdc[22799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:56 dist-test-slave-25dq krb5kdc[22799](info): set up 2 sockets
Sep 27 21:44:56 dist-test-slave-25dq krb5kdc[22799](info): commencing operation
krb5kdc: starting...
W20260927 21:44:57.892943 810 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:44:58 dist-test-slave-25dq krb5kdc[22799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:44:58.168246 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40067
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:40067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:44:58.503990 22812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:44:58.504369 22812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:44:58.504549 22812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:44:58.512032 22812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:44:58.512149 22812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:44:58.512243 22812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:44:58.512413 22812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:44:58.512485 22812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:44:58.522554 22812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:40067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:44:58.523975 22812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:44:58.525367 22812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:44:58.537473 22817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:58.537539 22820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:44:58.537576 22818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:44:58.538585 22812 server_base.cc:1034] running on GCE node
I20260927 21:44:58.539649 22812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:44:58.541272 22812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:44:58.542511 22812 hybrid_clock.cc:648] HybridClock initialized: now 1790545498542481 us; error 67 us; skew 500 ppm
I20260927 21:44:58.542958 22812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:44:58 dist-test-slave-25dq krb5kdc[22799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:44:58.554544 22812 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:44:58.556473 22812 webserver.cc:460] Webserver started at http://127.0.202.190:37495/ using document root <none> and password file <none>
I20260927 21:44:58.557219 22812 fs_manager.cc:359] Metadata directory not provided
I20260927 21:44:58.557350 22812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:44:58.557686 22812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:44:58.560859 22812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "5cb6a628303d409d9ec860a880a6b388"
format_stamp: "Formatted at 2026-09-27 21:44:58 on dist-test-slave-25dq"
server_key: "cdc7dc9cc00fb39e328c4c579f16749f"
server_key_iv: "479759a57da07496f47996651d4dece3"
server_key_version: "encryptionkey@0"
I20260927 21:44:58.561643 22812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "5cb6a628303d409d9ec860a880a6b388"
format_stamp: "Formatted at 2026-09-27 21:44:58 on dist-test-slave-25dq"
server_key: "cdc7dc9cc00fb39e328c4c579f16749f"
server_key_iv: "479759a57da07496f47996651d4dece3"
server_key_version: "encryptionkey@0"
I20260927 21:44:58.567150 22812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 21:44:58.570744 22827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:44:58.572255 22812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:44:58.572540 22812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "5cb6a628303d409d9ec860a880a6b388"
format_stamp: "Formatted at 2026-09-27 21:44:58 on dist-test-slave-25dq"
server_key: "cdc7dc9cc00fb39e328c4c579f16749f"
server_key_iv: "479759a57da07496f47996651d4dece3"
server_key_version: "encryptionkey@0"
I20260927 21:44:58.572808 22812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:44:58.578768 22812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:44:58.583845 22812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:44:58.584229 22812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:44:58.605170 22812 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:40067
I20260927 21:44:58.605206 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:40067 every 8 connection(s)
I20260927 21:44:58.606642 22812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:44:58.610571 22879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:44:58.616166 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22812
I20260927 21:44:58.616371 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:44:58.616932 810 external_mini_cluster.cc:1428] Setting key e7edf6b6ea2599b418a6667db53c5eb5
I20260927 21:44:58.621804 22879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388: Bootstrap starting.
I20260927 21:44:58.625007 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:58.618750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42607 (local address 127.0.202.190:40067)
0927 21:44:58.619627 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:58.619639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:58.619683 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:44:58.621378 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:58.621403 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:58.621747 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:58.622112 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:58.623222 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:58.623229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:58.623242 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:58.623262 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:58.623323 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:58.623345 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:58.623521 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:58.623665 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:44:58.623990 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":435,"threads_started":1}
I20260927 21:44:58.625600 22879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388: Neither blocks nor log segments found. Creating new log.
I20260927 21:44:58.626693 22879 log.cc:824] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388: Log is configured to *not* fsync() on all Append() calls
I20260927 21:44:58.629025 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:44:58.630085 22879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388: No bootstrap required, opened a new log
I20260927 21:44:58.638754 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:44:58.636762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52683 (local address 127.0.202.190:40067)
0927 21:44:58.636964 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:44:58.636972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:44:58.636994 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:44:58.637059 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:44:58.637064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:44:58.637131 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:44:58.637250 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:44:58.638159 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:44:58.638163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:44:58.638167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:44:58.638181 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:44:58.638227 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:44:58.638246 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:44:58.638343 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:44:58.638459 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 21:44:58.638511 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:44:58.641366 22879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb6a628303d409d9ec860a880a6b388" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40067 } }
I20260927 21:44:58.641764 22879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:44:58.641865 22879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb6a628303d409d9ec860a880a6b388, State: Initialized, Role: FOLLOWER
I20260927 21:44:58.642570 22879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb6a628303d409d9ec860a880a6b388" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40067 } }
I20260927 21:44:58.642791 22879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:44:58.642944 22879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:44:58.643116 22879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:44:58.645728 22879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb6a628303d409d9ec860a880a6b388" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40067 } }
I20260927 21:44:58.646245 22879 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb6a628303d409d9ec860a880a6b388; no voters:
I20260927 21:44:58.646855 22879 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:44:58.646987 22889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:44:58.647791 22889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb6a628303d409d9ec860a880a6b388, State: Running, Role: LEADER
I20260927 21:44:58.648418 22889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb6a628303d409d9ec860a880a6b388" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40067 } }
I20260927 21:44:58.649189 22879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:44:58.651427 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb6a628303d409d9ec860a880a6b388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb6a628303d409d9ec860a880a6b388" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40067 } } }
I20260927 21:44:58.651877 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:58.652362 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb6a628303d409d9ec860a880a6b388. Latest consensus state: current_term: 1 leader_uuid: "5cb6a628303d409d9ec860a880a6b388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb6a628303d409d9ec860a880a6b388" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 40067 } } }
I20260927 21:44:58.652794 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388 [sys.catalog]: This master's current role is: LEADER
I20260927 21:44:58.654259 22898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:44:58.660002 22898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:44:58.668814 22898 catalog_manager.cc:1347] Generated new cluster ID: 134a78b079a140e896a8ce6b8ef0acff
I20260927 21:44:58.668946 22898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:44:58.679447 22898 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:44:58.679934 22898 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:44:58.688794 22898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb6a628303d409d9ec860a880a6b388: Generated new TSK 0
I20260927 21:44:58.689848 22898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:44:58.703210 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22812
W20260927 21:44:58.704499 22874 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42607
W20260927 21:44:58.705168 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22812: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:44:58.705291 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22812
2026-09-27T21:44:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1948 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:44:58 dist-test-slave-25dq krb5kdc[22910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:44:58 dist-test-slave-25dq krb5kdc[22910](info): set up 2 sockets
Sep 27 21:44:58 dist-test-slave-25dq krb5kdc[22910](info): commencing operation
krb5kdc: starting...
W20260927 21:44:59.845432 810 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:45:00 dist-test-slave-25dq krb5kdc[22910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:45:00.120065 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:36645
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:36645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:45:00.458717 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:00.459115 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:00.459303 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:45:00.467092 22923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:00.467219 22923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:00.467306 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:00.467396 22923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:00.467455 22923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:00.477849 22923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:36645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:36645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:00.479215 22923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:00.480701 22923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:00.493413 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:00.493368 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:00.493513 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:00.494627 22923 server_base.cc:1034] running on GCE node
I20260927 21:45:00.495621 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:00.497269 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:00.498492 22923 hybrid_clock.cc:648] HybridClock initialized: now 1790545500498444 us; error 38 us; skew 500 ppm
I20260927 21:45:00.498920 22923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:00 dist-test-slave-25dq krb5kdc[22910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:00.502416 22923 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:45:00.504086 22923 webserver.cc:460] Webserver started at http://127.0.202.190:45667/ using document root <none> and password file <none>
I20260927 21:45:00.504957 22923 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:00.505095 22923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:00.505442 22923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:00.509317 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d"
format_stamp: "Formatted at 2026-09-27 21:45:00 on dist-test-slave-25dq"
server_key: "bd24667dbd5e5d32a2ded0a5fe71a5e4"
server_key_iv: "a7fa3632558c58130e320cb4e38a6d66"
server_key_version: "encryptionkey@0"
I20260927 21:45:00.510179 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d"
format_stamp: "Formatted at 2026-09-27 21:45:00 on dist-test-slave-25dq"
server_key: "bd24667dbd5e5d32a2ded0a5fe71a5e4"
server_key_iv: "a7fa3632558c58130e320cb4e38a6d66"
server_key_version: "encryptionkey@0"
I20260927 21:45:00.516132 22923 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260927 21:45:00.520083 22938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:00.521685 22923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:45:00.521854 22923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d"
format_stamp: "Formatted at 2026-09-27 21:45:00 on dist-test-slave-25dq"
server_key: "bd24667dbd5e5d32a2ded0a5fe71a5e4"
server_key_iv: "a7fa3632558c58130e320cb4e38a6d66"
server_key_version: "encryptionkey@0"
I20260927 21:45:00.522040 22923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:00.560109 22923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:00.564810 22923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:00.565116 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:00.584359 22923 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:36645
I20260927 21:45:00.584445 22989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:36645 every 8 connection(s)
I20260927 21:45:00.585767 22923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:45:00.587266 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 22923
I20260927 21:45:00.587446 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:45:00.587924 810 external_mini_cluster.cc:1428] Setting key 970e4c579774771888f4fa8fd45b8fce
I20260927 21:45:00.590255 22990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:45:00 dist-test-slave-25dq krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:45:00.603753 22990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d: Bootstrap starting.
I20260927 21:45:00.607264 22990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:00.608670 22990 log.cc:824] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:00.611312 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:00.589721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48239 (local address 127.0.202.190:36645)
0927 21:45:00.590509 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:00.590521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:00.590566 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:45:00.592167 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:00.592187 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:00.592654 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:00.592997 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:00.594843 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:00.596388 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:00.598142 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:00.598797 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:00.598918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:00.603674 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:00.603682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:00.603698 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:00.603741 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:00.606823 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:00.607550 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:00.607567 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:00.607576 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:00.607823 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:00.608189 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:00.608194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:00.608197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:00.608991 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:00.609361 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:00.609763 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 21:45:00.610180 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":351,"threads_started":1}
I20260927 21:45:00.611842 22990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d: No bootstrap required, opened a new log
I20260927 21:45:00.615763 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:45:00.622774 22990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36645 } }
I20260927 21:45:00.623149 22990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:45:00.623304 22990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110ebd2454fc41cbb57a8e0c3d5a2d4d, State: Initialized, Role: FOLLOWER
I20260927 21:45:00.624010 22990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36645 } }
I20260927 21:45:00.624344 22990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:45:00.624509 22990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:45:00.624694 22990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:00.627352 22990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36645 } }
I20260927 21:45:00.628001 22990 leader_election.cc:302] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110ebd2454fc41cbb57a8e0c3d5a2d4d; no voters:
I20260927 21:45:00.628960 22990 leader_election.cc:288] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:45:00.629134 23000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:45:00.630514 23000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [term 1 LEADER]: Becoming Leader. State: Replica: 110ebd2454fc41cbb57a8e0c3d5a2d4d, State: Running, Role: LEADER
I20260927 21:45:00.631207 23000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36645 } }
I20260927 21:45:00.631486 22990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:00.640986 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110ebd2454fc41cbb57a8e0c3d5a2d4d. Latest consensus state: current_term: 1 leader_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36645 } } }
I20260927 21:45:00.641212 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110ebd2454fc41cbb57a8e0c3d5a2d4d" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 36645 } } }
I20260927 21:45:00.641507 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:00.641539 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:00.642616 23007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:45:00.648723 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:00.621378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45993 (local address 127.0.202.190:36645)
0927 21:45:00.621582 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:00.621590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:00.622014 (+ 424us) server_negotiation.cc:407] Connection header received
0927 21:45:00.622181 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:00.622186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:00.622267 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:00.622381 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:00.623953 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:00.625121 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:00.626876 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:00.627549 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:00.627674 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:00.637875 (+ 10201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:00.637887 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:00.637895 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:00.637983 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:00.643379 (+ 5396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:00.644408 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:00.644415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:00.644418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:00.644700 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:00.645327 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:00.645335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:00.645341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:00.645906 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:00.647624 (+ 1718us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:00.647845 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:45:00.648161 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:45:00.649473 23007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:45:00.660676 23007 catalog_manager.cc:1347] Generated new cluster ID: f401154184524c078d1f4cd69e902a38
I20260927 21:45:00.660862 23007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:45:00.688158 23007 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:45:00.690318 23007 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:45:00.699687 23007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 110ebd2454fc41cbb57a8e0c3d5a2d4d: Generated new TSK 0
I20260927 21:45:00.700883 23007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:45:00.710036 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 22923
I20260927 21:45:00.711207 22985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:45:00.806952 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 22923
2026-09-27T21:45:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2104 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:45:00 dist-test-slave-25dq krb5kdc[23022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:45:00 dist-test-slave-25dq krb5kdc[23022](info): set up 2 sockets
Sep 27 21:45:00 dist-test-slave-25dq krb5kdc[23022](info): commencing operation
krb5kdc: starting...
W20260927 21:45:01.940971 810 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:45:02 dist-test-slave-25dq krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:45:02.224960 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41695
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:41695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:45:02.572000 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:02.572512 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:02.572702 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:45:02.584972 23035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:02.585140 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:02.585232 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:02.585333 23035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:02.585403 23035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:02.603678 23035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:41695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:02.605288 23035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:02.606839 23035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:02.621275 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:02.621410 23043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:02.622289 23041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:02.622879 23035 server_base.cc:1034] running on GCE node
I20260927 21:45:02.623734 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:02.625346 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:02.626619 23035 hybrid_clock.cc:648] HybridClock initialized: now 1790545502626523 us; error 59 us; skew 500 ppm
I20260927 21:45:02.627082 23035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:02 dist-test-slave-25dq krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:02.631785 23035 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:45:02.634095 23035 webserver.cc:460] Webserver started at http://127.0.202.190:35597/ using document root <none> and password file <none>
I20260927 21:45:02.635274 23035 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:02.635443 23035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:02.635905 23035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:02.640826 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "f62b2e90d57d44c5b131456ac806ae32"
format_stamp: "Formatted at 2026-09-27 21:45:02 on dist-test-slave-25dq"
server_key: "5e0f1e66671dfe1339a4eccae583637a"
server_key_iv: "dc1847a947b27659a903f63b461e5b6f"
server_key_version: "encryptionkey@0"
I20260927 21:45:02.641703 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "f62b2e90d57d44c5b131456ac806ae32"
format_stamp: "Formatted at 2026-09-27 21:45:02 on dist-test-slave-25dq"
server_key: "5e0f1e66671dfe1339a4eccae583637a"
server_key_iv: "dc1847a947b27659a903f63b461e5b6f"
server_key_version: "encryptionkey@0"
I20260927 21:45:02.647773 23035 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 21:45:02.652079 23050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:02.654249 23035 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:45:02.654503 23035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "f62b2e90d57d44c5b131456ac806ae32"
format_stamp: "Formatted at 2026-09-27 21:45:02 on dist-test-slave-25dq"
server_key: "5e0f1e66671dfe1339a4eccae583637a"
server_key_iv: "dc1847a947b27659a903f63b461e5b6f"
server_key_version: "encryptionkey@0"
I20260927 21:45:02.654740 23035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:02.694732 23035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:02.699579 23035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:02.699926 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:02.723049 23035 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:41695
I20260927 21:45:02.723071 23101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:41695 every 8 connection(s)
I20260927 21:45:02.724764 23035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:45:02.728928 23102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:02.732884 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23035
I20260927 21:45:02.733057 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:45:02.733609 810 external_mini_cluster.cc:1428] Setting key 7425344c4d37d439138ec6e0cfa94950
I20260927 21:45:02.742324 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32: Bootstrap starting.
I20260927 21:45:02.746124 23102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:02.747376 23102 log.cc:824] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:02.748612 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:02.735589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41677 (local address 127.0.202.190:41695)
0927 21:45:02.736443 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:02.736456 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:02.736499 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:45:02.738236 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:02.738259 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:02.738691 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:02.739035 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:02.739679 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:02.741311 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:02.743139 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:02.743984 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:02.744145 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:02.745560 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:02.745571 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:02.745587 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:45:02.746273 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:02.746376 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:45:02.746412 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:02.746542 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:02.746776 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:45:02.747069 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":401,"threads_started":1}
I20260927 21:45:02.750725 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32: No bootstrap required, opened a new log
I20260927 21:45:02.752645 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:45:02.762518 23102 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62b2e90d57d44c5b131456ac806ae32" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41695 } }
I20260927 21:45:02.762996 23102 raft_consensus.cc:374] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:45:02.763139 23102 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62b2e90d57d44c5b131456ac806ae32, State: Initialized, Role: FOLLOWER
I20260927 21:45:02.763926 23102 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62b2e90d57d44c5b131456ac806ae32" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41695 } }
I20260927 21:45:02.764261 23102 raft_consensus.cc:388] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:45:02.764462 23102 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:45:02.764675 23102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:02.766239 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:02.758520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51511 (local address 127.0.202.190:41695)
0927 21:45:02.758777 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:02.758783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:02.759295 (+ 512us) server_negotiation.cc:407] Connection header received
0927 21:45:02.759479 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:02.759486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:02.759579 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:02.759741 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:02.760406 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:02.761668 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:02.763029 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:02.763749 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:02.763884 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:02.765335 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:02.765342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:02.765347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:45:02.765377 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:02.765435 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:45:02.765465 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:02.765576 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:02.765752 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 21:45:02.765868 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 21:45:02.768159 23102 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62b2e90d57d44c5b131456ac806ae32" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41695 } }
I20260927 21:45:02.768754 23102 leader_election.cc:302] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f62b2e90d57d44c5b131456ac806ae32; no voters:
I20260927 21:45:02.769400 23102 leader_election.cc:288] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:45:02.769595 23112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:45:02.770375 23112 raft_consensus.cc:686] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [term 1 LEADER]: Becoming Leader. State: Replica: f62b2e90d57d44c5b131456ac806ae32, State: Running, Role: LEADER
I20260927 21:45:02.771057 23112 consensus_queue.cc:237] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62b2e90d57d44c5b131456ac806ae32" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41695 } }
I20260927 21:45:02.771665 23102 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:02.774111 23113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f62b2e90d57d44c5b131456ac806ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62b2e90d57d44c5b131456ac806ae32" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41695 } } }
I20260927 21:45:02.775223 23113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:02.775059 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f62b2e90d57d44c5b131456ac806ae32. Latest consensus state: current_term: 1 leader_uuid: "f62b2e90d57d44c5b131456ac806ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62b2e90d57d44c5b131456ac806ae32" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 41695 } } }
I20260927 21:45:02.775900 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32 [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:02.778839 23118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:45:02.784839 23118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:45:02.794739 23118 catalog_manager.cc:1347] Generated new cluster ID: cdfe9291295540ff9dfc4cc73b535e1f
I20260927 21:45:02.794867 23118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:45:02.821949 23118 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:45:02.823480 23118 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:45:02.836823 23118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f62b2e90d57d44c5b131456ac806ae32: Generated new TSK 0
I20260927 21:45:02.837754 23118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:45:02.847136 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23035
W20260927 21:45:02.848428 23097 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41677
W20260927 21:45:02.849193 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23035: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:45:02.849382 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23035
2026-09-27T21:45:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:45:02 dist-test-slave-25dq krb5kdc[23133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:45:02 dist-test-slave-25dq krb5kdc[23133](info): set up 2 sockets
Sep 27 21:45:02 dist-test-slave-25dq krb5kdc[23133](info): commencing operation
krb5kdc: starting...
W20260927 21:45:04.049419 810 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:45:04 dist-test-slave-25dq krb5kdc[23133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:45:04.328831 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33151
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:45:04.669942 23146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:04.670253 23146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:04.670428 23146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:45:04.677937 23146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:04.678061 23146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:04.678166 23146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:04.678275 23146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:04.678340 23146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:04.688649 23146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:04.690045 23146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:04.691373 23146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:04.703559 23152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:04.703804 23154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:04.704010 23151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:04.704555 23146 server_base.cc:1034] running on GCE node
I20260927 21:45:04.705385 23146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:04.706866 23146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:04.708093 23146 hybrid_clock.cc:648] HybridClock initialized: now 1790545504707995 us; error 94 us; skew 500 ppm
I20260927 21:45:04.708514 23146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:04 dist-test-slave-25dq krb5kdc[23133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:04.719149 23146 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:45:04.721117 23146 webserver.cc:460] Webserver started at http://127.0.202.190:34359/ using document root <none> and password file <none>
I20260927 21:45:04.721910 23146 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:04.722031 23146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:04.722329 23146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:04.726168 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "ed4a69c4cd9e47ca87b9247e90345eeb"
format_stamp: "Formatted at 2026-09-27 21:45:04 on dist-test-slave-25dq"
server_key: "e5b6851d7e8a2fa7a42ed451c0d66a01"
server_key_iv: "a444626953991413c6794056c57e1064"
server_key_version: "encryptionkey@0"
I20260927 21:45:04.727075 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "ed4a69c4cd9e47ca87b9247e90345eeb"
format_stamp: "Formatted at 2026-09-27 21:45:04 on dist-test-slave-25dq"
server_key: "e5b6851d7e8a2fa7a42ed451c0d66a01"
server_key_iv: "a444626953991413c6794056c57e1064"
server_key_version: "encryptionkey@0"
I20260927 21:45:04.733415 23146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 21:45:04.737733 23161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:04.739600 23146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 21:45:04.739943 23146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "ed4a69c4cd9e47ca87b9247e90345eeb"
format_stamp: "Formatted at 2026-09-27 21:45:04 on dist-test-slave-25dq"
server_key: "e5b6851d7e8a2fa7a42ed451c0d66a01"
server_key_iv: "a444626953991413c6794056c57e1064"
server_key_version: "encryptionkey@0"
I20260927 21:45:04.740191 23146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:04.747162 23146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:04.752523 23146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:04.752902 23146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:04.774616 23146 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:33151
I20260927 21:45:04.774731 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:33151 every 8 connection(s)
I20260927 21:45:04.776435 23146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:45:04.780661 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:04.785336 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23146
I20260927 21:45:04.785576 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:45:04.786134 810 external_mini_cluster.cc:1428] Setting key cf9caf3754a0058d8e04fe7beafc402b
I20260927 21:45:04.792578 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb: Bootstrap starting.
I20260927 21:45:04.794728 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:04.788034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45585 (local address 127.0.202.190:33151)
0927 21:45:04.788921 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:04.788934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:04.788977 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:45:04.791127 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:04.791152 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:04.791530 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:04.791930 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:04.793073 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:04.793079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:04.793093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:45:04.793109 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:04.793167 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:45:04.793187 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:04.793433 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:04.793575 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:45:04.793774 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":420,"threads_started":1}
I20260927 21:45:04.796339 23213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:04.797627 23213 log.cc:824] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:04.799933 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:45:04.802827 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb: No bootstrap required, opened a new log
I20260927 21:45:04.808485 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:04.804608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34727 (local address 127.0.202.190:33151)
0927 21:45:04.804848 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:04.804855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:04.805233 (+ 378us) server_negotiation.cc:407] Connection header received
0927 21:45:04.805386 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:04.805398 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:04.805474 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:04.805612 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:04.807354 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:04.807363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:04.807370 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:45:04.807397 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:04.807472 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:45:04.807504 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:04.807634 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:04.807976 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 21:45:04.808091 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:45:04.813140 23213 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33151 } }
I20260927 21:45:04.813613 23213 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:45:04.813791 23213 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4a69c4cd9e47ca87b9247e90345eeb, State: Initialized, Role: FOLLOWER
I20260927 21:45:04.814522 23213 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33151 } }
I20260927 21:45:04.814805 23213 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:45:04.814976 23213 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:45:04.815183 23213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:04.817809 23213 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33151 } }
I20260927 21:45:04.818418 23213 leader_election.cc:302] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed4a69c4cd9e47ca87b9247e90345eeb; no voters:
I20260927 21:45:04.819240 23213 leader_election.cc:288] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:45:04.819404 23223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:45:04.820104 23223 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [term 1 LEADER]: Becoming Leader. State: Replica: ed4a69c4cd9e47ca87b9247e90345eeb, State: Running, Role: LEADER
I20260927 21:45:04.820822 23223 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33151 } }
I20260927 21:45:04.821851 23213 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:04.824005 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed4a69c4cd9e47ca87b9247e90345eeb. Latest consensus state: current_term: 1 leader_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33151 } } }
I20260927 21:45:04.824460 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:04.824503 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4a69c4cd9e47ca87b9247e90345eeb" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33151 } } }
I20260927 21:45:04.824908 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:04.831956 23230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:45:04.837601 23230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:45:04.846675 23230 catalog_manager.cc:1347] Generated new cluster ID: 8b21de3f30b447498d275eed6c6f00da
I20260927 21:45:04.846810 23230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:45:04.860021 23230 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:45:04.860642 23230 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:45:04.879069 23230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed4a69c4cd9e47ca87b9247e90345eeb: Generated new TSK 0
I20260927 21:45:04.880344 23230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:45:04.889180 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23146
W20260927 21:45:04.890399 23208 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45585
W20260927 21:45:04.891171 810 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23146: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:45:04.891343 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23146
2026-09-27T21:45:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2041 ms)
[----------] 18 tests from AuthTokenIssuingTest (36676 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:45:05 dist-test-slave-25dq krb5kdc[23244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:45:05 dist-test-slave-25dq krb5kdc[23244](info): set up 2 sockets
Sep 27 21:45:05 dist-test-slave-25dq krb5kdc[23244](info): commencing operation
krb5kdc: starting...
W20260927 21:45:06.013952 810 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:45:06 dist-test-slave-25dq krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:45:06.297207 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:43187
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:43187,127.0.202.189:37255,127.0.202.188:36457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:45:06.642711 23257 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:45:06.643055 23257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:06.643126 23257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:06.643210 23257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:45:06.650763 23257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:06.650880 23257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:06.650939 23257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:06.650983 23257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:06.651046 23257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:06.661350 23257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:43187,127.0.202.189:37255,127.0.202.188:36457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:43187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:06.662647 23257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:06.664093 23257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:06.676748 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:06.676805 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:06.676854 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:06.677855 23257 server_base.cc:1034] running on GCE node
I20260927 21:45:06.678642 23257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:06.680083 23257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:06.681318 23257 hybrid_clock.cc:648] HybridClock initialized: now 1790545506681229 us; error 110 us; skew 500 ppm
I20260927 21:45:06.681710 23257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:06 dist-test-slave-25dq krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:06.685104 23257 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:45:06.686688 23257 webserver.cc:460] Webserver started at http://127.0.202.190:37563/ using document root <none> and password file <none>
I20260927 21:45:06.687350 23257 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:06.687433 23257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:06.687708 23257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:06.690845 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "417c159fa0914bbe8de82b730cc1dc76"
format_stamp: "Formatted at 2026-09-27 21:45:06 on dist-test-slave-25dq"
server_key: "33b14f1cd6d197f613f6d1b5fc30a385"
server_key_iv: "239ba4ca7260ca65a57cfe18667e0d72"
server_key_version: "encryptionkey@0"
I20260927 21:45:06.691614 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "417c159fa0914bbe8de82b730cc1dc76"
format_stamp: "Formatted at 2026-09-27 21:45:06 on dist-test-slave-25dq"
server_key: "33b14f1cd6d197f613f6d1b5fc30a385"
server_key_iv: "239ba4ca7260ca65a57cfe18667e0d72"
server_key_version: "encryptionkey@0"
I20260927 21:45:06.697770 23257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260927 21:45:06.701597 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:06.703186 23257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:45:06.703420 23257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "417c159fa0914bbe8de82b730cc1dc76"
format_stamp: "Formatted at 2026-09-27 21:45:06 on dist-test-slave-25dq"
server_key: "33b14f1cd6d197f613f6d1b5fc30a385"
server_key_iv: "239ba4ca7260ca65a57cfe18667e0d72"
server_key_version: "encryptionkey@0"
I20260927 21:45:06.703693 23257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:06.728397 23257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:06.733444 23257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:06.733735 23257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:06.753450 23257 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:43187
I20260927 21:45:06.753481 23323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:43187 every 8 connection(s)
I20260927 21:45:06.754973 23257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:45:06.758841 23324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:06.765355 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23257
I20260927 21:45:06.765581 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:45:06.766374 810 external_mini_cluster.cc:1428] Setting key 199b6536fcfbbddc39dcfb9fd61a89af
I20260927 21:45:06.765760 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:06 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:45:06.793924 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.770027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:43187 (local address 127.0.0.1:50070)
0927 21:45:06.770959 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:06.771015 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:06.773313 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:06.774408 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:06.774414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:06.775200 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:06.775790 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:06.775821 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:06.778349 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:06.778353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:06.779501 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:06.779522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:06.780924 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:06.780929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:06.781095 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:06.781860 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:06.781888 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:06.784689 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:06.789544 (+ 4855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:06.789559 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:06.789574 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:06.790020 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:06.790554 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:06.790558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:06.790560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:06.790759 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:06.791884 (+ 1125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:06.791890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:06.792077 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 21:45:06.792416 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 21:45:06.792839 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":421,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.189@KRBTEST.COM; defaulting to no policy
I20260927 21:45:06.793964 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.771463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.0.202.190:43187)
0927 21:45:06.772222 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:06.772233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:06.772316 (+ 83us) server_negotiation.cc:407] Connection header received
0927 21:45:06.773769 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:06.773780 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:06.774127 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:06.774372 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:06.776060 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:06.778154 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:06.779675 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:06.780669 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:06.780870 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:06.784967 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:06.784981 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:06.785002 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:06.785047 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:06.789354 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:06.790172 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:06.790192 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:06.790196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:06.790419 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:06.790879 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:06.790886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:06.790891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:06.791714 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:06.791920 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:06.792587 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 21:45:06.793057 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":502,"thread_start_us":360,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.189@KRBTEST.COM" created.
I20260927 21:45:06.797729 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:45:06.799291 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.798696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48574)
0927 21:45:06.798956 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:06.799105 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 21:45:06.799778 23275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
I20260927 21:45:06.801746 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.801229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48584)
0927 21:45:06.801428 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:06.801530 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 21:45:06.802004 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:37255: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab.
Entry for principal kudu/127.0.202.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab.
I20260927 21:45:06.832763 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:37255
--webserver_interface=127.0.202.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab
--principal=kudu/127.0.202.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:43187,127.0.202.189:37255,127.0.202.188:36457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:45:06.851389 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } attempt: 1
I20260927 21:45:06.852751 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.852229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48588)
0927 21:45:06.852480 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:06.852601 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:45:06.853941 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.853522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48592)
0927 21:45:06.853700 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:06.853797 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 21:45:06.854149 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:37255: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
I20260927 21:45:06.922429 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } attempt: 2
I20260927 21:45:06.923714 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.923208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48594)
0927 21:45:06.923481 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:06.923568 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 21:45:06.924865 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:06.924475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48604)
0927 21:45:06.924659 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:06.924732 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 21:45:06.925051 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:37255: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
I20260927 21:45:07.016381 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } attempt: 3
I20260927 21:45:07.017506 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.017062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48614)
0927 21:45:07.017269 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.017369 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 21:45:07.018601 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.018237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48616)
0927 21:45:07.018384 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.018459 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:45:07.018775 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:37255: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
I20260927 21:45:07.161994 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } attempt: 4
I20260927 21:45:07.163272 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.162654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48622)
0927 21:45:07.162975 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.163097 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 21:45:07.164376 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.163948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48630)
0927 21:45:07.164099 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.164193 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:45:07.164685 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:37255: Network error: Client connection negotiation failed: client connection to 127.0.202.189:37255: connect: Connection refused (error 111)
W20260927 21:45:07.166468 23331 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:45:07.166812 23331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:07.166906 23331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:07.167047 23331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:45:07.174546 23331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:07.174634 23331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:07.174728 23331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:07.174811 23331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:07.174896 23331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:07.184916 23331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:43187,127.0.202.189:37255,127.0.202.188:36457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab
--principal=kudu/127.0.202.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:37255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.202.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:07.186257 23331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:07.187599 23331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:07.199730 23336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:07.199685 23339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:07.199887 23337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:07.201732 23331 server_base.cc:1034] running on GCE node
I20260927 21:45:07.202524 23331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:07.203994 23331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:07.205264 23331 hybrid_clock.cc:648] HybridClock initialized: now 1790545507205174 us; error 68 us; skew 500 ppm
I20260927 21:45:07.205621 23331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:07.209250 23331 init.cc:377] Logged in from keytab as kudu/127.0.202.189@KRBTEST.COM (short username kudu)
I20260927 21:45:07.210796 23331 webserver.cc:460] Webserver started at http://127.0.202.189:44721/ using document root <none> and password file <none>
I20260927 21:45:07.211503 23331 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:07.211628 23331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:07.211970 23331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:07.215229 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data/instance:
uuid: "70a750a0705a46de9aab414043d37cc9"
format_stamp: "Formatted at 2026-09-27 21:45:07 on dist-test-slave-25dq"
server_key: "0e6b4aa6023339e1038eb5d255a278ce"
server_key_iv: "09bbfd106b3f14212c786fb8c54c24dc"
server_key_version: "encryptionkey@0"
I20260927 21:45:07.216096 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/wal/instance:
uuid: "70a750a0705a46de9aab414043d37cc9"
format_stamp: "Formatted at 2026-09-27 21:45:07 on dist-test-slave-25dq"
server_key: "0e6b4aa6023339e1038eb5d255a278ce"
server_key_iv: "09bbfd106b3f14212c786fb8c54c24dc"
server_key_version: "encryptionkey@0"
I20260927 21:45:07.221835 23331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:45:07.225611 23346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:07.227203 23331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 21:45:07.227407 23331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/wal
uuid: "70a750a0705a46de9aab414043d37cc9"
format_stamp: "Formatted at 2026-09-27 21:45:07 on dist-test-slave-25dq"
server_key: "0e6b4aa6023339e1038eb5d255a278ce"
server_key_iv: "09bbfd106b3f14212c786fb8c54c24dc"
server_key_version: "encryptionkey@0"
I20260927 21:45:07.227629 23331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:07.253194 23331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:07.257922 23331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:07.258242 23331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:07.277909 23331 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.189:37255
I20260927 21:45:07.277941 23397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.189:37255 every 8 connection(s)
I20260927 21:45:07.279332 23331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
I20260927 21:45:07.283224 23398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:07.289435 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23331
I20260927 21:45:07.289654 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-1/wal/instance
I20260927 21:45:07.290094 810 external_mini_cluster.cc:1428] Setting key 2441608c281913cb29a49ff87f8852e4
I20260927 21:45:07.289943 23398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:45:07.312487 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.292767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.0.202.190:43187)
0927 21:45:07.293019 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.293027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.294249 (+ 1222us) server_negotiation.cc:407] Connection header received
0927 21:45:07.296174 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.296184 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.296286 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.296433 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.298048 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.298730 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.300228 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.300704 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.300842 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.304731 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.304741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.304749 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.304797 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.308879 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.309783 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.309791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.309797 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.309950 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.310372 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.310378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.310383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.310815 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.310965 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.311722 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.312001 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 21:45:07.313199 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.293237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:43187 (local address 127.0.0.1:50086)
0927 21:45:07.294063 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.294110 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:07.295891 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:07.296726 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:07.296735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:07.297291 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:07.297863 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.297881 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.298897 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.298901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.300077 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.300089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.300928 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.300935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.301169 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.301876 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:07.301899 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:07.304476 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:07.309104 (+ 4628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.309111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.309126 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.309650 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.310075 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.310080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.310084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.310257 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.310955 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:07.310961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:07.311306 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 21:45:07.311548 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 21:45:07.311973 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":367,"threads_started":1}
WARNING: no policy specified for kudu/127.0.202.188@KRBTEST.COM; defaulting to no policy
I20260927 21:45:07.315131 23398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.188@KRBTEST.COM" created.
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:07.332563 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.316038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48644)
0927 21:45:07.316256 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.316274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:07.316458 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:07.317567 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:07.317573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:07.318125 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:07.318308 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.318321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.319692 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.319700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.321237 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.321254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.322140 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.322152 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.322361 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.322985 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:07.323011 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:07.325521 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:07.329266 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.329274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.329281 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.329851 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.330249 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.330256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.330263 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.330447 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.331139 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:07.331146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:07.331325 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 21:45:07.331477 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 21:45:07.331974 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 21:45:07.332628 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.316364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.0.202.189:37255)
0927 21:45:07.317116 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.317122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.317147 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:45:07.317190 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.317196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.317374 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.317493 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.318496 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.319542 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.321383 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.321983 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.322082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.325746 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.325763 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.325786 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.325828 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.329122 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.329998 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.330005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.330007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.330125 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.330591 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.330595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.330597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.330964 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.331224 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.331942 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.332208 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":394,"threads_started":1}
I20260927 21:45:07.335296 23398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:45:07.336544 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.336027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52116)
0927 21:45:07.336189 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.336387 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 21:45:07.336812 23349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
I20260927 21:45:07.338416 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.338008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52122)
0927 21:45:07.338231 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.338310 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 21:45:07.338644 23398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab.
Entry for principal kudu/127.0.202.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab.
I20260927 21:45:07.356072 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:36457
--webserver_interface=127.0.202.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab
--principal=kudu/127.0.202.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:43187,127.0.202.189:37255,127.0.202.188:36457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:45:07.387979 23398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 1
I20260927 21:45:07.389200 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.388724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52126)
0927 21:45:07.388919 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.389032 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:45:07.390368 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.389950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52132)
0927 21:45:07.390155 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.390251 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 21:45:07.390570 23398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
I20260927 21:45:07.458809 23398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 2
I20260927 21:45:07.460052 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.459582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52144)
0927 21:45:07.459778 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.459868 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:45:07.461143 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.460760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52158)
0927 21:45:07.460935 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.461016 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 21:45:07.461335 23398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
I20260927 21:45:07.464008 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } attempt: 5
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:07.479965 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.464793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48656)
0927 21:45:07.465007 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.465030 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:07.465173 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:07.465580 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:07.465584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:07.466139 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:07.466386 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.466398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.467454 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.467462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.469151 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.469169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.470012 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.470018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.470267 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.470970 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:07.470995 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:07.473629 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:07.477215 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.477221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.477226 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.477652 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.478080 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.478085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.478089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.478269 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.478877 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:07.478883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:07.479060 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 21:45:07.479228 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 21:45:07.479555 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:45:07.479961 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.464862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.0.202.189:37255)
0927 21:45:07.465081 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.465086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.465110 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:45:07.465333 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.465337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.465416 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.465521 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.466561 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.467284 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.469330 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.469854 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.469953 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.473843 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.473848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.473851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.473886 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.477042 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.477799 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.477808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.477817 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.477951 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.478405 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.478411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.478415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.478742 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.478917 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.479350 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.479585 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 21:45:07.481180 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:45:07.482245 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.481841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52170)
0927 21:45:07.482022 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.482103 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:45:07.483204 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.482861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52186)
0927 21:45:07.483011 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.483092 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:45:07.483404 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
I20260927 21:45:07.534634 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 1
I20260927 21:45:07.535841 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.535364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52194)
0927 21:45:07.535568 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.535672 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:45:07.537027 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.536650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52196)
0927 21:45:07.536818 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.536897 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 21:45:07.537377 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
I20260927 21:45:07.552584 23398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 3
I20260927 21:45:07.553977 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.553322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52206)
0927 21:45:07.553598 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.553730 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 21:45:07.555414 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.554812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52208)
0927 21:45:07.555106 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.555226 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260927 21:45:07.555739 23398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
I20260927 21:45:07.605681 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 2
I20260927 21:45:07.606873 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.606393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52224)
0927 21:45:07.606595 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.606709 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 21:45:07.608045 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.607657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52232)
0927 21:45:07.607843 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.607922 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 21:45:07.608316 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
W20260927 21:45:07.697985 23405 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:45:07.698387 23405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:07.698518 23405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:07.698661 23405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:45:07.698977 23398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 4
I20260927 21:45:07.700199 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.699634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52248)
0927 21:45:07.699888 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.699995 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:45:07.701243 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.700944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52262)
0927 21:45:07.701071 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.701142 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:45:07.701493 23398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
W20260927 21:45:07.706808 23405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:07.706898 23405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:07.707007 23405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:07.707086 23405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:07.707198 23405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:07.714557 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 3
I20260927 21:45:07.715891 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.715281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52270)
0927 21:45:07.715620 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.715723 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 21:45:07.717062 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.716685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52286)
0927 21:45:07.716877 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.716959 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 21:45:07.717314 23324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:36457: Network error: Client connection negotiation failed: client connection to 127.0.202.188:36457: connect: Connection refused (error 111)
I20260927 21:45:07.718273 23405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:43187,127.0.202.189:37255,127.0.202.188:36457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab
--principal=kudu/127.0.202.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.202.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:07.719638 23405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:07.721083 23405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:07.733865 23411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:07.734266 23413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:07.733865 23410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:07.736351 23405 server_base.cc:1034] running on GCE node
I20260927 21:45:07.737146 23405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:07.738631 23405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:07.739851 23405 hybrid_clock.cc:648] HybridClock initialized: now 1790545507739789 us; error 34 us; skew 500 ppm
I20260927 21:45:07.740209 23405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:07.743444 23405 init.cc:377] Logged in from keytab as kudu/127.0.202.188@KRBTEST.COM (short username kudu)
I20260927 21:45:07.745034 23405 webserver.cc:460] Webserver started at http://127.0.202.188:44451/ using document root <none> and password file <none>
I20260927 21:45:07.745710 23405 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:07.745796 23405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:07.746062 23405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:07.749027 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data/instance:
uuid: "790f69daac514e6dbc9c1cf201bd7328"
format_stamp: "Formatted at 2026-09-27 21:45:07 on dist-test-slave-25dq"
server_key: "e9625f542440b91b411688673b6af564"
server_key_iv: "cee6b8f6cce0defc17792b9731cf97e9"
server_key_version: "encryptionkey@0"
I20260927 21:45:07.749795 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/wal/instance:
uuid: "790f69daac514e6dbc9c1cf201bd7328"
format_stamp: "Formatted at 2026-09-27 21:45:07 on dist-test-slave-25dq"
server_key: "e9625f542440b91b411688673b6af564"
server_key_iv: "cee6b8f6cce0defc17792b9731cf97e9"
server_key_version: "encryptionkey@0"
I20260927 21:45:07.755295 23405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:45:07.759313 23420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:07.760967 23405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:45:07.761166 23405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/wal
uuid: "790f69daac514e6dbc9c1cf201bd7328"
format_stamp: "Formatted at 2026-09-27 21:45:07 on dist-test-slave-25dq"
server_key: "e9625f542440b91b411688673b6af564"
server_key_iv: "cee6b8f6cce0defc17792b9731cf97e9"
server_key_version: "encryptionkey@0"
I20260927 21:45:07.761433 23405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:07.778530 23405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:07.783648 23405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:07.783972 23405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:07.805281 23405 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.188:36457
I20260927 21:45:07.805318 23471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.188:36457 every 8 connection(s)
I20260927 21:45:07.806893 23405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
I20260927 21:45:07.811275 23472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:07.812737 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23405
I20260927 21:45:07.812937 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790545135802652-810-0/minicluster-data/master-2/wal/instance
I20260927 21:45:07.813537 810 external_mini_cluster.cc:1428] Setting key c348757e0e6a93316b3ca24d1140df4e
I20260927 21:45:07.821684 23472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:45:07.838229 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.815839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.202.188:36457)
0927 21:45:07.816707 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.816721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.816764 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:45:07.818495 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.818520 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.818993 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.819432 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.821022 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.822340 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.823706 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.824515 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.824746 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.829633 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.829643 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.829655 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.829694 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.833545 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.834387 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.834401 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.834412 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.834632 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.835024 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.835035 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.835041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.835729 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.835961 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.836543 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.836911 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":401,"threads_started":1}
I20260927 21:45:07.842190 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.824703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:43187 (local address 127.0.0.1:50098)
0927 21:45:07.826068 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.826150 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:07.826404 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:07.826829 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:07.826836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:07.827494 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:07.827790 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.827803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.829237 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.829244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.830433 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.830445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.831201 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.831207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.831439 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.832139 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:07.832156 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:07.835248 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:07.839042 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.839051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.839058 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.839648 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.840098 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.840101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.840104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.840229 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.840924 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:07.840935 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:07.841173 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 21:45:07.841374 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 21:45:07.841694 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":452,"threads_started":1}
I20260927 21:45:07.842231 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.824707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.0.202.190:43187)
0927 21:45:07.825812 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.825819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.826302 (+ 483us) server_negotiation.cc:407] Connection header received
0927 21:45:07.826551 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.826561 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.826635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.826762 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.827976 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.829053 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.830581 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.830999 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.831149 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.835449 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.835454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.835458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.835492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.838900 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.839795 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.839800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.839803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.839971 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.840396 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.840401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.840404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.840760 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.840925 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.841523 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.841776 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":839,"threads_started":1}
I20260927 21:45:07.844121 23472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:07.858567 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.841866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.0.202.190:43187)
0927 21:45:07.842502 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.842511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.842533 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:45:07.842607 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.842615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.842728 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.842851 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.844332 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.845135 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.846801 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.847306 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.847482 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.852604 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.852609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.852613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.852646 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.855544 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.856403 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.856407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.856409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.856524 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.856954 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.856972 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.856975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.857265 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.857567 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.857921 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.858180 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":989,"threads_started":1}
I20260927 21:45:07.859256 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.844963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.0.202.189:37255)
0927 21:45:07.845164 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.845176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.845218 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:45:07.845374 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.845381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.845474 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.845584 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.846352 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.847260 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.848788 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.849619 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.849773 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.853391 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.853399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.853406 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.853458 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.856685 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.857305 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.857311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.857314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.857481 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.857867 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.857873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.857877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.858135 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.858301 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.858665 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.858886 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:45:07.859633 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.844936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:37255 (local address 127.0.0.1:48666)
0927 21:45:07.845118 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.845135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:07.845232 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:07.845679 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:07.845683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:07.846035 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:07.846192 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.846200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.847417 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.847424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.848634 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.848648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.849796 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.849801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.849956 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.850577 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:07.850608 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:07.853186 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:07.856814 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.856818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.856821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.857171 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.857615 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.857626 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.857629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.857757 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.858263 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:07.858268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:07.858420 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 21:45:07.858572 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 21:45:07.859291 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:45:07.860749 23472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:07.874631 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.861395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52300)
0927 21:45:07.861660 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.861677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:07.861777 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:07.862047 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:07.862051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:07.862429 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:07.862574 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.862582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.863529 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.863537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.864425 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.864435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.865209 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.865215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.865414 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.866067 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:07.866084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:07.868471 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:07.871770 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.871777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.871781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.872324 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.872805 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.872812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.872817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.872981 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.873652 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:07.873657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:07.873830 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 21:45:07.873984 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 21:45:07.874315 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:45:07.875111 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.861665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.0.202.188:36457)
0927 21:45:07.861813 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.861820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.861838 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:45:07.861877 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.861882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.861940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.862034 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.862703 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.863410 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.864550 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.865041 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.865188 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.868697 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.868702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.868705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.868742 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.871615 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.872445 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.872448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.872470 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.872618 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.873125 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.873131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.873134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.873518 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.873627 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.874086 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.874515 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":59}
I20260927 21:45:07.876940 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.860559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.0.202.189:37255)
0927 21:45:07.860725 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.860729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.861357 (+ 628us) server_negotiation.cc:407] Connection header received
0927 21:45:07.861542 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.861549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.861627 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.861733 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.862910 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.863627 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.864860 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.865323 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.865454 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.870832 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.870838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.870841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.870876 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.874210 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.875046 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.875050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.875053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.875195 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.875647 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.875652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.875655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.875920 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.876204 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.876492 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.876676 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:45:07.880860 23472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: Bootstrap starting.
I20260927 21:45:07.881789 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:45:07.884178 23472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:07.885560 23472 log.cc:824] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:07.884421 23299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "417c159fa0914bbe8de82b730cc1dc76"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50108
I20260927 21:45:07.888896 23472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: No bootstrap required, opened a new log
I20260927 21:45:07.894636 23324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 4
I20260927 21:45:07.899510 23472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:07.899982 23472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790f69daac514e6dbc9c1cf201bd7328, State: Initialized, Role: FOLLOWER
I20260927 21:45:07.900604 23472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:07.901847 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:07.902252 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:07.903151 23472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 21:45:07 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
I20260927 21:45:07.914206 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.895521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52314)
0927 21:45:07.895827 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:07.895850 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:07.895970 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:07.896482 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:07.896487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:07.896998 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:07.897218 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.897231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.898986 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.898992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.900498 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:07.900514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.901717 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.901722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:07.901906 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.902686 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:07.902719 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:07.905614 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:07.910776 (+ 5162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.910783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.910788 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.911235 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.911973 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:07.911981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:07.911988 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:07.912180 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:07.913022 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:07.913030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:07.913238 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 21:45:07.913376 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 21:45:07.913725 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 21:45:07.915485 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:07.895680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.0.202.188:36457)
0927 21:45:07.895909 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:07.895915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:07.895938 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:45:07.896139 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:07.896161 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:07.896287 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:07.896448 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:07.897404 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.898804 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.900651 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:07.901552 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:07.901705 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:07.905876 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:07.905883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:07.905887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:07.905930 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:07.910617 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.911379 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.911385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.911389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.911540 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:07.912346 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:07.912351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:07.912355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:07.912834 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:07.914657 (+ 1823us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:07.914796 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 21:45:07.915054 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:45:07.921963 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76: Bootstrap starting.
I20260927 21:45:07.911947 23405 master_runner.cc:186] Error getting master registration for 127.0.202.190:43187: OK, instance_id { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" instance_seqno: 1790545506747644 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:45:07.925215 23405 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:45:07.926496 23324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:07.927923 23324 log.cc:824] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76: Log is configured to *not* fsync() on all Append() calls
W20260927 21:45:07.930572 23491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:45:07.930706 23491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:45:07.931648 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76: No bootstrap required, opened a new log
I20260927 21:45:07.940251 23324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:07.940750 23324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417c159fa0914bbe8de82b730cc1dc76, State: Initialized, Role: FOLLOWER
I20260927 21:45:07.941321 23324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:07.942360 23494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:07.942791 23494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:07.943768 23324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:07.943856 23299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:45:07.944056 23299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:07.948660 23299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:07.951542 23299 leader_election.cc:288] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [CANDIDATE]: Term 1 election: Requested vote from peers 70a750a0705a46de9aab414043d37cc9 (127.0.202.189:37255), 790f69daac514e6dbc9c1cf201bd7328 (127.0.202.188:36457)
I20260927 21:45:07.953433 23447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "417c159fa0914bbe8de82b730cc1dc76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "790f69daac514e6dbc9c1cf201bd7328"
I20260927 21:45:07.954082 23447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:07.959424 23447 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417c159fa0914bbe8de82b730cc1dc76 in term 1.
I20260927 21:45:07.960454 23275 leader_election.cc:302] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417c159fa0914bbe8de82b730cc1dc76, 790f69daac514e6dbc9c1cf201bd7328; no voters:
I20260927 21:45:07.961256 23494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:45:07.963307 23494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [term 1 LEADER]: Becoming Leader. State: Replica: 417c159fa0914bbe8de82b730cc1dc76, State: Running, Role: LEADER
I20260927 21:45:07.963979 23494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:07.964721 23257 master_runner.cc:186] Error getting master registration for 127.0.202.189:37255: OK, instance_id { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" instance_seqno: 1790545507272244 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:45:07.983873 23257 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:45:07.985185 23505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:45:07.985335 23505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:45:07.985519 23508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417c159fa0914bbe8de82b730cc1dc76. Latest consensus state: current_term: 1 leader_uuid: "417c159fa0914bbe8de82b730cc1dc76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:07.986020 23508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: This master's current role is: LEADER
W20260927 21:45:07.986725 23278 tablet.cc:2369] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:45:08.000697 23398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } attempt: 5
I20260927 21:45:08.003489 23512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:45:08.005867 23512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:45:08 dist-test-slave-25dq krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
I20260927 21:45:08.026222 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:08.002419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.0.202.188:36457)
0927 21:45:08.002632 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.002641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.002665 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:45:08.002745 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.002751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.002854 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.002966 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.004510 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.006025 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.008460 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.009423 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.009578 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:08.016066 (+ 6488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:08.016079 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:08.016088 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:08.016155 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:08.021580 (+ 5425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.023134 (+ 1554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.023144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.023151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.023379 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.024371 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.024377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.024380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.024793 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:08.024959 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:08.025577 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 21:45:08.025826 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:45:08.026247 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:08.001494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:36457 (local address 127.0.0.1:52318)
0927 21:45:08.001973 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:08.001999 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:08.002156 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:08.003217 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:08.003225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:08.003892 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:08.004198 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:08.004216 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.006283 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.006291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:08.008214 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:08.008234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.009860 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.009869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:08.010193 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:08.011101 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:08.011147 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:08.015706 (+ 4559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:08.021969 (+ 6263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:08.021979 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:08.021986 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:08.022900 (+ 914us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:08.023958 (+ 1058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:08.023964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:08.023967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:08.024181 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:08.025044 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:08.025049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:08.025243 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 21:45:08.025429 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 21:45:08.025792 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311}
I20260927 21:45:08.033361 23398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: Bootstrap starting.
W20260927 21:45:08.036486 23426 tablet.cc:2369] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:45:08.038050 23398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:08.039567 23398 log.cc:824] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:08.055665 23398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: No bootstrap required, opened a new log
I20260927 21:45:08.069221 23398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:08.069846 23398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a750a0705a46de9aab414043d37cc9, State: Initialized, Role: FOLLOWER
I20260927 21:45:08.070556 23398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } }
I20260927 21:45:08.078109 23519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:08.080909 23519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:08.080773 23398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:08.081508 23373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "417c159fa0914bbe8de82b730cc1dc76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "70a750a0705a46de9aab414043d37cc9"
I20260927 21:45:08.082067 23373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:08.087261 23373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417c159fa0914bbe8de82b730cc1dc76 in term 1.
I20260927 21:45:08.109931 23447 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [term 1 FOLLOWER]: Refusing update from remote peer 417c159fa0914bbe8de82b730cc1dc76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:45:08.111519 23508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:45:08.122247 23373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [term 1 FOLLOWER]: Refusing update from remote peer 417c159fa0914bbe8de82b730cc1dc76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:45:08.123574 23508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:45:08.133347 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:08.025766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.0.202.189:37255)
0927 21:45:08.025967 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.025972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.038552 (+ 12580us) server_negotiation.cc:407] Connection header received
0927 21:45:08.039115 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.039125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.039247 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.039358 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.044646 (+ 5288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.046289 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.057450 (+ 11161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.058307 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.058435 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:08.097951 (+ 39516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:08.097963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:08.097972 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:08.098055 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:08.103586 (+ 5531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.106683 (+ 3097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.106695 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.106702 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.106997 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.126970 (+ 19973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.126980 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.126987 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.127553 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:08.127877 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:08.132340 (+ 4463us) server_negotiation.cc:299] Negotiation successful
0927 21:45:08.132703 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:45:08.147008 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417c159fa0914bbe8de82b730cc1dc76. Latest consensus state: current_term: 1 leader_uuid: "417c159fa0914bbe8de82b730cc1dc76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:08.147526 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:08.147419 23519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417c159fa0914bbe8de82b730cc1dc76. Latest consensus state: current_term: 1 leader_uuid: "417c159fa0914bbe8de82b730cc1dc76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:08.147742 23519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:08.155860 23514 mvcc.cc:205] Tried to move back new op lower bound from 7334074401142419456 to 7334074400626606080. Current Snapshot: MvccSnapshot[applied={T|T < 7334074401142419456}]
I20260927 21:45:08.163975 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "417c159fa0914bbe8de82b730cc1dc76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:08.164500 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:08.165729 23508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "417c159fa0914bbe8de82b730cc1dc76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:08.166144 23508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:08.166692 23508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "417c159fa0914bbe8de82b730cc1dc76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:08.166997 23508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76 [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:08.168481 23512 catalog_manager.cc:1347] Generated new cluster ID: 809cc2bee78346ddb33a2e87b2ddcee5
I20260927 21:45:08.168589 23512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:45:08.175527 23519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "417c159fa0914bbe8de82b730cc1dc76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c159fa0914bbe8de82b730cc1dc76" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 43187 } } peers { permanent_uuid: "70a750a0705a46de9aab414043d37cc9" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 37255 } } peers { permanent_uuid: "790f69daac514e6dbc9c1cf201bd7328" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 36457 } } }
I20260927 21:45:08.175830 23519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:08.193549 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:08.026114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.0.202.190:43187)
0927 21:45:08.026305 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.026313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.029922 (+ 3609us) server_negotiation.cc:407] Connection header received
0927 21:45:08.031409 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.031420 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.031537 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.031629 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.066189 (+ 34560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.067549 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.095089 (+ 27540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.096218 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.096390 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:08.138549 (+ 42159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:08.138556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:08.138561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:08.138632 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:08.144133 (+ 5501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.167179 (+ 23046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.167188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.167194 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.167411 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.183139 (+ 15728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.183163 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.183171 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.183846 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:08.184353 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:08.187544 (+ 3191us) server_negotiation.cc:299] Negotiation successful
0927 21:45:08.187972 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:45:08.199208 23537 catalog_manager.cc:1259] Loaded cluster ID: 809cc2bee78346ddb33a2e87b2ddcee5
I20260927 21:45:08.199390 23537 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: loading cluster ID for follower catalog manager: success
I20260927 21:45:08.199658 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:08.032635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.0.202.188:36457)
0927 21:45:08.040271 (+ 7636us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.040278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.054934 (+ 14656us) server_negotiation.cc:407] Connection header received
0927 21:45:08.056893 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.056900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.056992 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.057085 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.080012 (+ 22927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.081272 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.125100 (+ 43828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:08.126106 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:08.126217 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:08.177126 (+ 50909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:08.177171 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:08.177181 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:08.177288 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:08.182897 (+ 5609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.186910 (+ 4013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.186921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.186928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.187168 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:08.189830 (+ 2662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:08.189836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:08.189840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:08.190285 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:08.190571 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:08.199012 (+ 8441us) server_negotiation.cc:299] Negotiation successful
0927 21:45:08.199302 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7480}
W20260927 21:45:08.204447 23537 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260927 21:45:08.205626 23537 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:45:08.231467 23512 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:45:08.233662 23512 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:45:08.258893 23512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 417c159fa0914bbe8de82b730cc1dc76: Generated new TSK 0
I20260927 21:45:08.260150 23512 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:45:08.386798 23476 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.371294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.0.202.190:43187)
0927 21:45:08.371562 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.371573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.384893 (+ 13320us) server_negotiation.cc:407] Connection header received
0927 21:45:08.385205 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.385212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.385298 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.385377 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.386544 (+ 1167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50126: BlockingRecv error: recv got EOF from 127.0.0.1:50126 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 21:45:08.389169 23475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.385089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.0.202.188:36457)
0927 21:45:08.385268 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.385274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.387574 (+ 2300us) server_negotiation.cc:407] Connection header received
0927 21:45:08.387838 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.387844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.387928 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.387999 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.388942 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52328: BlockingRecv error: recv got EOF from 127.0.0.1:52328 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 21:45:08.389441 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.375966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.0.202.189:37255)
0927 21:45:08.376191 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.376198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.387249 (+ 11051us) server_negotiation.cc:407] Connection header received
0927 21:45:08.387908 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.387914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.388025 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.388106 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.389209 (+ 1103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48708: BlockingRecv error: recv got EOF from 127.0.0.1:48708 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 21:45:08.433126 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.422247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.0.202.189:37255)
0927 21:45:08.422453 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.422460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.430552 (+ 8092us) server_negotiation.cc:407] Connection header received
0927 21:45:08.431176 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.431183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.431341 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.431465 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.432718 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48720: BlockingRecv error: recv got EOF from 127.0.0.1:48720 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 21:45:08.479375 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.465148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.0.202.189:37255)
0927 21:45:08.465367 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.465374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.477031 (+ 11657us) server_negotiation.cc:407] Connection header received
0927 21:45:08.477276 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.477283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.477402 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.477538 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.479112 (+ 1574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48722: BlockingRecv error: recv got EOF from 127.0.0.1:48722 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 21:45:08.530021 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.520050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.0.202.189:37255)
0927 21:45:08.520276 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.520283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.527775 (+ 7492us) server_negotiation.cc:407] Connection header received
0927 21:45:08.528326 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.528333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.528458 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.528545 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.529689 (+ 1144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48738: BlockingRecv error: recv got EOF from 127.0.0.1:48738 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 21:45:08.595207 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.573178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.0.202.189:37255)
0927 21:45:08.573381 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.573389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.593025 (+ 19636us) server_negotiation.cc:407] Connection header received
0927 21:45:08.593261 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.593270 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.593379 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.593522 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.594929 (+ 1407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48746: BlockingRecv error: recv got EOF from 127.0.0.1:48746 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 21:45:08.647614 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.643229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.202.189:37255)
0927 21:45:08.643469 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.643476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.646083 (+ 2607us) server_negotiation.cc:407] Connection header received
0927 21:45:08.646195 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.646201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.646307 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.646438 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.647334 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48750: BlockingRecv error: recv got EOF from 127.0.0.1:48750 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 21:45:08.673131 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.670551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.0.202.189:37255)
0927 21:45:08.670779 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.670787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.671381 (+ 594us) server_negotiation.cc:407] Connection header received
0927 21:45:08.671635 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.671642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.671737 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.671846 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.672858 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48766: BlockingRecv error: recv got EOF from 127.0.0.1:48766 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 21:45:08.711795 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.709255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.0.202.189:37255)
0927 21:45:08.709595 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.709602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.709950 (+ 348us) server_negotiation.cc:407] Connection header received
0927 21:45:08.710162 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.710166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.710239 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.710345 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.711372 (+ 1027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48776: BlockingRecv error: recv got EOF from 127.0.0.1:48776 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 21:45:08.785125 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.782681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.0.202.189:37255)
0927 21:45:08.782916 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.782927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.783516 (+ 589us) server_negotiation.cc:407] Connection header received
0927 21:45:08.783766 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.783780 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.783857 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.783987 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.784841 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48782: BlockingRecv error: recv got EOF from 127.0.0.1:48782 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 21:45:08.919130 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:08.916760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.202.189:37255)
0927 21:45:08.917072 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:08.917079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:08.917412 (+ 333us) server_negotiation.cc:407] Connection header received
0927 21:45:08.917714 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:08.917719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:08.917791 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:08.917904 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:08.918870 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48784: BlockingRecv error: recv got EOF from 127.0.0.1:48784 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 21:45:08.932871 23491 catalog_manager.cc:1259] Loaded cluster ID: 809cc2bee78346ddb33a2e87b2ddcee5
I20260927 21:45:08.932972 23491 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: loading cluster ID for follower catalog manager: success
I20260927 21:45:08.935783 23491 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: acquiring CA information for follower catalog manager: success
I20260927 21:45:08.937753 23491 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 790f69daac514e6dbc9c1cf201bd7328: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 21:45:09.184265 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:09.182350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.0.202.189:37255)
0927 21:45:09.182640 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:09.182647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:09.182856 (+ 209us) server_negotiation.cc:407] Connection header received
0927 21:45:09.183018 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:09.183022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:09.183101 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:09.183194 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:09.184043 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50316: BlockingRecv error: recv got EOF from 127.0.0.1:50316 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 21:45:09.209107 23537 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: acquiring CA information for follower catalog manager: success
I20260927 21:45:09.211000 23537 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70a750a0705a46de9aab414043d37cc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:45:09.708847 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:09.702093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.202.189:37255)
0927 21:45:09.702638 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:09.702643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:09.702731 (+ 88us) server_negotiation.cc:407] Connection header received
0927 21:45:09.702992 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:09.702997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:09.703175 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:09.703293 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:09.703780 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.704705 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.706156 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.706763 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.706945 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:09.707436 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:09.708107 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:09.708255 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:09.708395 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:45:09.708470 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":288,"threads_started":1}
I20260927 21:45:09.723234 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:09.716223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.0.202.188:36457)
0927 21:45:09.716945 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:09.716950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:09.716972 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:45:09.717076 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:09.717080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:09.717245 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:09.717450 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:09.718008 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.718801 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.720665 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.721145 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.721267 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:09.721839 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:09.722480 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:09.722673 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:09.722888 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 21:45:09.722970 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":500,"threads_started":1}
I20260927 21:45:09.741479 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:09.733373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.0.202.189:37255)
0927 21:45:09.733622 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:09.733633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:09.735279 (+ 1646us) server_negotiation.cc:407] Connection header received
0927 21:45:09.735494 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:09.735515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:09.735601 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:09.735711 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:09.736179 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.737103 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.738980 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.739680 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.739796 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:09.740541 (+ 745us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:09.740706 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:09.740774 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:09.741027 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 21:45:09.741128 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:45:09.748587 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:09.734389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.0.202.188:36457)
0927 21:45:09.734592 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:09.734598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:09.742238 (+ 7640us) server_negotiation.cc:407] Connection header received
0927 21:45:09.742373 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:09.742380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:09.742494 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:09.742607 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:09.743103 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.744272 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.746201 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.746950 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.747106 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:09.747643 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:09.747765 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:09.747890 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:09.748127 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 21:45:09.748224 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:45:09.752131 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:09.733331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.0.202.190:43187)
0927 21:45:09.745765 (+ 12434us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:09.745771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:09.745794 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:45:09.745924 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:09.745929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:09.746123 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:09.746282 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:09.746925 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.747702 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.749920 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:09.750354 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:09.750457 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:09.750943 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:09.751311 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:09.751420 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:09.751756 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 21:45:09.751858 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12271,"thread_start_us":12168,"threads_started":1}
I20260927 21:45:09.762534 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23257
I20260927 21:45:09.763681 23319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:45:09.879838 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23257
W20260927 21:45:09.903198 23422 connection.cc:535] server connection from 127.0.0.1:52314 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 21:45:09.903529 23349 connection.cc:535] server connection from 127.0.0.1:48656 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:45:09.903733 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23331
I20260927 21:45:09.905130 23393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:45:10.014557 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23331
I20260927 21:45:10.036360 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23405
I20260927 21:45:10.037496 23467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:45:10.145871 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23405
2026-09-27T21:45:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5258 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:45:10 dist-test-slave-25dq krb5kdc[23624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:45:10 dist-test-slave-25dq krb5kdc[23624](info): set up 2 sockets
Sep 27 21:45:10 dist-test-slave-25dq krb5kdc[23624](info): commencing operation
krb5kdc: starting...
W20260927 21:45:11.277626 810 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/test-user.keytab.
Sep 27 21:45:11 dist-test-slave-25dq krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:45:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
Entry for principal kudu/127.0.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab.
I20260927 21:45:11.544600 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33563
--webserver_interface=127.0.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33563,127.0.202.189:46629,127.0.202.188:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:45:11.886857 23637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:45:11.887252 23637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:11.887379 23637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:11.887527 23637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:45:11.894922 23637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:11.895064 23637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:11.895174 23637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:11.895262 23637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:11.895375 23637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:11.905916 23637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33563,127.0.202.189:46629,127.0.202.188:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.190.keytab
--principal=kudu/127.0.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.190:33563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:11.907320 23637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:11.908774 23637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:11.921787 23642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:11.921834 23645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:11.921875 23643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:11.922885 23637 server_base.cc:1034] running on GCE node
I20260927 21:45:11.923678 23637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:11.925186 23637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:11.926409 23637 hybrid_clock.cc:648] HybridClock initialized: now 1790545511926348 us; error 29 us; skew 500 ppm
I20260927 21:45:11.926821 23637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:11 dist-test-slave-25dq krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:11.930279 23637 init.cc:377] Logged in from keytab as kudu/127.0.202.190@KRBTEST.COM (short username kudu)
I20260927 21:45:11.931785 23637 webserver.cc:460] Webserver started at http://127.0.202.190:45853/ using document root <none> and password file <none>
I20260927 21:45:11.932574 23637 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:11.932705 23637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:11.933022 23637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:11.935998 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data/instance:
uuid: "135fcf8d390a427cbe53606bcb457067"
format_stamp: "Formatted at 2026-09-27 21:45:11 on dist-test-slave-25dq"
server_key: "39f5f89c27dad67094c705ad2379b46a"
server_key_iv: "e344fc648d99684a35acd2673b7d4bda"
server_key_version: "encryptionkey@0"
I20260927 21:45:11.937042 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/wal/instance:
uuid: "135fcf8d390a427cbe53606bcb457067"
format_stamp: "Formatted at 2026-09-27 21:45:11 on dist-test-slave-25dq"
server_key: "39f5f89c27dad67094c705ad2379b46a"
server_key_iv: "e344fc648d99684a35acd2673b7d4bda"
server_key_version: "encryptionkey@0"
I20260927 21:45:11.942433 23637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:45:11.946190 23652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:11.947573 23637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:45:11.947798 23637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/wal
uuid: "135fcf8d390a427cbe53606bcb457067"
format_stamp: "Formatted at 2026-09-27 21:45:11 on dist-test-slave-25dq"
server_key: "39f5f89c27dad67094c705ad2379b46a"
server_key_iv: "e344fc648d99684a35acd2673b7d4bda"
server_key_version: "encryptionkey@0"
I20260927 21:45:11.948019 23637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:11.963389 23637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:11.968251 23637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:11.968654 23637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:11.989799 23637 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.190:33563
I20260927 21:45:11.989779 23703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.190:33563 every 8 connection(s)
I20260927 21:45:11.991364 23637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/data/info.pb
I20260927 21:45:11.995245 23704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:12.001427 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23637
I20260927 21:45:12.001634 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-0/wal/instance
I20260927 21:45:12.002159 810 external_mini_cluster.cc:1428] Setting key 13dfd2b60df0fc5abeed2f8709539e40
I20260927 21:45:12.002246 23704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:12 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.189@KRBTEST.COM; defaulting to no policy
I20260927 21:45:12.030488 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.006579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.0.202.190:33563)
0927 21:45:12.008559 (+ 1980us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:12.008565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:12.008608 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:45:12.010998 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:12.011009 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:12.011357 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:12.011571 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:12.013421 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.015418 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.017032 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.018034 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.018243 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.022160 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:12.022170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:12.022186 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:12.022220 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:12.025645 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.026656 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.026663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.026665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.026819 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.027377 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.027381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.027384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.027962 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:12.028138 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:12.028928 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 21:45:12.029288 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1797,"spinlock_wait_cycles":34048,"thread_start_us":1516,"threads_started":1}
I20260927 21:45:12.030598 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.006579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33563 (local address 127.0.0.1:50174)
0927 21:45:12.007522 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.007580 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:12.010549 (+ 2969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:12.011662 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:12.011670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:12.012604 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:12.013154 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.013175 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.015590 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.015594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.016885 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.016898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.018272 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.018277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.018411 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.019140 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:12.019164 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:12.021857 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:12.025803 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.025819 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.025834 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.026486 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.026950 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.026957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.026963 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.027208 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.028132 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:12.028143 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:12.028456 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 21:45:12.028755 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 21:45:12.029519 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":157,"thread_start_us":409,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.189@KRBTEST.COM" created.
I20260927 21:45:12.033820 23704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:45:12.035022 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.034611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35096)
0927 21:45:12.034793 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.034912 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 21:45:12.035312 23655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
I20260927 21:45:12.036994 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.036606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35108)
0927 21:45:12.036790 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.036868 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 21:45:12.037190 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:46629: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab.
Entry for principal kudu/127.0.202.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab.
I20260927 21:45:12.064884 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:46629
--webserver_interface=127.0.202.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab
--principal=kudu/127.0.202.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33563,127.0.202.189:46629,127.0.202.188:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:45:12.086557 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } attempt: 1
I20260927 21:45:12.087786 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.087258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35120)
0927 21:45:12.087461 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.087657 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:45:12.088835 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.088505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35132)
0927 21:45:12.088649 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.088721 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 21:45:12.089039 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:46629: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
I20260927 21:45:12.157354 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } attempt: 2
I20260927 21:45:12.158638 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.158131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35140)
0927 21:45:12.158394 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.158490 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 21:45:12.159646 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.159294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35154)
0927 21:45:12.159434 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.159515 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 21:45:12.159893 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:46629: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
I20260927 21:45:12.251201 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } attempt: 3
I20260927 21:45:12.252765 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.252149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35162)
0927 21:45:12.252482 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.252629 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260927 21:45:12.253814 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.253427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35176)
0927 21:45:12.253617 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.253687 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 21:45:12.254067 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:46629: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
W20260927 21:45:12.396014 23711 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:45:12.396342 23711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:12.396482 23711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:12.396615 23711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:45:12.397306 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } attempt: 4
I20260927 21:45:12.398622 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.398032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35188)
0927 21:45:12.398348 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.398436 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 21:45:12.399809 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.399387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35202)
0927 21:45:12.399623 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.399693 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260927 21:45:12.400053 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.189:46629: Network error: Client connection negotiation failed: client connection to 127.0.202.189:46629: connect: Connection refused (error 111)
W20260927 21:45:12.404683 23711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:12.404763 23711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:12.404861 23711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:12.404939 23711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:12.405026 23711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:12.415132 23711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33563,127.0.202.189:46629,127.0.202.188:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.189.keytab
--principal=kudu/127.0.202.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.189:46629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.202.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:12.416589 23711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:12.417955 23711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:12.430045 23716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:12.429996 23717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:12.429945 23719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:12.430805 23711 server_base.cc:1034] running on GCE node
I20260927 21:45:12.431578 23711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:12.433233 23711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:12.434535 23711 hybrid_clock.cc:648] HybridClock initialized: now 1790545512434446 us; error 60 us; skew 500 ppm
I20260927 21:45:12.434888 23711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:12 dist-test-slave-25dq krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:12.443645 23711 init.cc:377] Logged in from keytab as kudu/127.0.202.189@KRBTEST.COM (short username kudu)
I20260927 21:45:12.445277 23711 webserver.cc:460] Webserver started at http://127.0.202.189:45461/ using document root <none> and password file <none>
I20260927 21:45:12.445995 23711 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:12.446132 23711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:12.446457 23711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:12.449524 23711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data/instance:
uuid: "039e06be2332495684361766e8b5baea"
format_stamp: "Formatted at 2026-09-27 21:45:12 on dist-test-slave-25dq"
server_key: "025cb9b7b3e262da1267544b391b8634"
server_key_iv: "f198186a17482c208701e0fa5856ddfe"
server_key_version: "encryptionkey@0"
I20260927 21:45:12.450313 23711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/wal/instance:
uuid: "039e06be2332495684361766e8b5baea"
format_stamp: "Formatted at 2026-09-27 21:45:12 on dist-test-slave-25dq"
server_key: "025cb9b7b3e262da1267544b391b8634"
server_key_iv: "f198186a17482c208701e0fa5856ddfe"
server_key_version: "encryptionkey@0"
I20260927 21:45:12.455542 23711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:45:12.459005 23726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:12.460443 23711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:45:12.460625 23711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/wal
uuid: "039e06be2332495684361766e8b5baea"
format_stamp: "Formatted at 2026-09-27 21:45:12 on dist-test-slave-25dq"
server_key: "025cb9b7b3e262da1267544b391b8634"
server_key_iv: "f198186a17482c208701e0fa5856ddfe"
server_key_version: "encryptionkey@0"
I20260927 21:45:12.460846 23711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:12.488858 23711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:12.493722 23711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:12.494028 23711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:12.513501 23711 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.189:46629
I20260927 21:45:12.513518 23777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.189:46629 every 8 connection(s)
I20260927 21:45:12.514976 23711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/data/info.pb
I20260927 21:45:12.518857 23778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:12.520663 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23711
I20260927 21:45:12.520844 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-1/wal/instance
I20260927 21:45:12.521384 810 external_mini_cluster.cc:1428] Setting key 2876939d99c848f0384d7e611331ac1e
I20260927 21:45:12.528831 23778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:12 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.202.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.202.188@KRBTEST.COM" created.
I20260927 21:45:12.551713 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.532413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.0.202.190:33563)
0927 21:45:12.533012 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:12.533016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:12.534068 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 21:45:12.535931 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:12.535937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:12.536030 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:12.536155 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:12.537731 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.538453 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.539967 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.540468 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.540630 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.544826 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:12.544831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:12.544834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:12.544871 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:12.547895 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.548852 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.548856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.548858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.548996 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.549538 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.549546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.549549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.549868 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:12.550015 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:12.551036 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0927 21:45:12.551281 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":390,"threads_started":1}
I20260927 21:45:12.552920 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.533147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33563 (local address 127.0.0.1:50190)
0927 21:45:12.533911 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.533950 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:12.535667 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:12.536423 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:12.536431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:12.536951 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:12.537514 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.537531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.538631 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.538635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.539819 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.539831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.540759 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.540767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.540993 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.541710 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:12.541736 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:12.544526 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:12.548052 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.548059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.548073 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.548701 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.549157 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.549162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.549164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.549385 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.549992 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:12.550000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:12.550543 (+ 543us) client_negotiation.cc:769] Sending connection context
0927 21:45:12.550928 (+ 385us) client_negotiation.cc:240] Negotiation successful
0927 21:45:12.551505 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":311,"threads_started":1}
I20260927 21:45:12.554911 23778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:12 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:12.570523 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.555877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.202.189:46629)
0927 21:45:12.556587 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:12.556594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:12.556621 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:45:12.556667 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:12.556673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:12.556861 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:12.556978 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:12.558014 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.559102 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.560314 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.560848 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.560987 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.564390 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:12.564402 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:12.564416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:12.564448 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:12.567509 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.568359 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.568370 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.568380 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.568494 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.568912 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.568915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.568917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.569263 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:12.569413 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:12.569899 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 21:45:12.570219 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":550,"thread_start_us":341,"threads_started":1}
I20260927 21:45:12.570605 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.555855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35210)
0927 21:45:12.556092 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.556114 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:12.556225 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:12.557025 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:12.557036 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:12.557561 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:12.557789 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.557803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.559253 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.559261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.560138 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.560149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.561025 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.561031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.561198 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.561800 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:12.561817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:12.564122 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:12.567645 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.567654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.567661 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.568184 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.568624 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.568632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.568638 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.568807 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.569407 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:12.569415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:12.569598 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 21:45:12.569757 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 21:45:12.570101 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 21:45:12.573428 23778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:45:12.574945 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.574308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59768)
0927 21:45:12.574545 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.574677 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 21:45:12.575250 23730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
I20260927 21:45:12.576936 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.576534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59770)
0927 21:45:12.576675 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.576758 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:45:12.577186 23778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.202.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab.
Entry for principal kudu/127.0.202.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab.
I20260927 21:45:12.588205 810 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Z3URy/build/asan/bin/kudu
/tmp/dist-test-task3Z3URy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:46013
--webserver_interface=127.0.202.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab
--principal=kudu/127.0.202.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.202.148:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.202.190:33563,127.0.202.189:46629,127.0.202.188:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:45:12.626500 23778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 1
I20260927 21:45:12.627837 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.627169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59774)
0927 21:45:12.627520 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.627661 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:45:12.629014 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.628567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59788)
0927 21:45:12.628720 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.628890 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:45:12.629292 23778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
I20260927 21:45:12.697508 23778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 2
I20260927 21:45:12.698730 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.698242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59804)
0927 21:45:12.698443 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.698574 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:45:12.699297 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } attempt: 5
I20260927 21:45:12.699965 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.699504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59814)
0927 21:45:12.699682 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.699781 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 21:45:12.700227 23778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Sep 27 21:45:12 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:12.714215 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.699863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35214)
0927 21:45:12.700115 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.700133 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:12.700253 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:12.700694 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:12.700699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:12.701195 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:12.701425 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.701438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.702464 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.702469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.703863 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:12.703879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.704714 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.704723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:12.704926 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.705579 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:12.705605 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:12.708128 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:12.711403 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.711409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.711413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.711828 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.712236 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:12.712246 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:12.712250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:12.712465 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:12.713164 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:12.713169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:12.713344 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 21:45:12.713501 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 21:45:12.713858 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 21:45:12.714213 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.700075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.0.202.189:46629)
0927 21:45:12.700331 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:12.700336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:12.700356 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:45:12.700436 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:12.700441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:12.700532 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:12.700616 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:12.701606 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.702324 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.704017 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:12.704550 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:12.704648 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:12.708389 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:12.708394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:12.708398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:12.708432 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:12.711227 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.711962 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.711969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.711972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.712114 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:12.712696 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:12.712704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:12.712713 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:12.713033 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:12.713203 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:12.713645 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 21:45:12.713889 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 21:45:12.715945 23704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:45:12.717348 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.716874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59828)
0927 21:45:12.717089 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.717192 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 21:45:12.718554 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.718136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59832)
0927 21:45:12.718333 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.718425 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 21:45:12.718930 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
I20260927 21:45:12.770236 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 1
I20260927 21:45:12.771636 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.771088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59844)
0927 21:45:12.771309 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.771408 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 21:45:12.772778 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.772456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59848)
0927 21:45:12.772599 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.772670 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:45:12.773012 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
I20260927 21:45:12.791551 23778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 3
I20260927 21:45:12.792912 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.792339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59856)
0927 21:45:12.792644 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.792741 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 21:45:12.794095 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.793646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59866)
0927 21:45:12.793891 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.793976 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 21:45:12.794308 23778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
I20260927 21:45:12.841270 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 2
I20260927 21:45:12.842669 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.842052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59872)
0927 21:45:12.842316 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.842489 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:45:12.843777 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.843379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59884)
0927 21:45:12.843585 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.843658 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 21:45:12.843988 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
I20260927 21:45:12.937584 23778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 4
I20260927 21:45:12.938869 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.938321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59898)
0927 21:45:12.938626 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.938715 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 21:45:12.939925 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.939591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59906)
0927 21:45:12.939759 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.939823 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:45:12.940145 23778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
I20260927 21:45:12.950188 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 3
I20260927 21:45:12.951354 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.950830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59922)
0927 21:45:12.951106 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.951188 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 21:45:12.952523 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:12.952089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59932)
0927 21:45:12.952343 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:12.952417 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260927 21:45:12.952723 23704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.202.188:46013: Network error: Client connection negotiation failed: client connection to 127.0.202.188:46013: connect: Connection refused (error 111)
W20260927 21:45:12.954795 23786 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:45:12.955185 23786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:45:12.955309 23786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:45:12.955474 23786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:45:12.963551 23786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:45:12.963682 23786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:45:12.963802 23786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:45:12.963894 23786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:45:12.963979 23786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:45:12.974663 23786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.202.148:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.202.190:33563,127.0.202.189:46629,127.0.202.188:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/krb5kdc/kudu_127.0.202.188.keytab
--principal=kudu/127.0.202.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.202.188:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.202.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:45:12.976069 23786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:45:12.977545 23786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:45:12.989724 23791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:12.989842 23792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:45:12.990131 23794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:45:12.992419 23786 server_base.cc:1034] running on GCE node
I20260927 21:45:12.993343 23786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:45:12.995108 23786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:45:12.996479 23786 hybrid_clock.cc:648] HybridClock initialized: now 1790545512996275 us; error 43 us; skew 500 ppm
I20260927 21:45:12.996868 23786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:45:12 dist-test-slave-25dq krb5kdc[23624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:45:13.000180 23786 init.cc:377] Logged in from keytab as kudu/127.0.202.188@KRBTEST.COM (short username kudu)
I20260927 21:45:13.001750 23786 webserver.cc:460] Webserver started at http://127.0.202.188:37995/ using document root <none> and password file <none>
I20260927 21:45:13.002452 23786 fs_manager.cc:359] Metadata directory not provided
I20260927 21:45:13.002547 23786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:45:13.002812 23786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:45:13.005954 23786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data/instance:
uuid: "0875acdf678b4371a04966c52b14be55"
format_stamp: "Formatted at 2026-09-27 21:45:12 on dist-test-slave-25dq"
server_key: "cb276e5723942b1f52e07e0109f956f0"
server_key_iv: "d6829a78308401f831dbc4450110de25"
server_key_version: "encryptionkey@0"
I20260927 21:45:13.006704 23786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/wal/instance:
uuid: "0875acdf678b4371a04966c52b14be55"
format_stamp: "Formatted at 2026-09-27 21:45:12 on dist-test-slave-25dq"
server_key: "cb276e5723942b1f52e07e0109f956f0"
server_key_iv: "d6829a78308401f831dbc4450110de25"
server_key_version: "encryptionkey@0"
I20260927 21:45:13.011955 23786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:45:13.015733 23801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:45:13.017531 23786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:45:13.017741 23786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data,/tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/wal
uuid: "0875acdf678b4371a04966c52b14be55"
format_stamp: "Formatted at 2026-09-27 21:45:12 on dist-test-slave-25dq"
server_key: "cb276e5723942b1f52e07e0109f956f0"
server_key_iv: "d6829a78308401f831dbc4450110de25"
server_key_version: "encryptionkey@0"
I20260927 21:45:13.017997 23786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:45:13.052083 23786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:45:13.057672 23786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:45:13.058005 23786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:45:13.079048 23786 rpc_server.cc:307] RPC server started. Bound to: 127.0.202.188:46013
I20260927 21:45:13.079093 23852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.202.188:46013 every 8 connection(s)
I20260927 21:45:13.080606 23786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/data/info.pb
I20260927 21:45:13.084776 23853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:45:13.084776 810 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Z3URy/build/asan/bin/kudu as pid 23786
I20260927 21:45:13.084942 810 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Z3URy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790545135802652-810-0/minicluster-data/master-2/wal/instance
I20260927 21:45:13.085492 810 external_mini_cluster.cc:1428] Setting key e10d447d09be013578ca542b23d37cda
I20260927 21:45:13.094281 23853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
I20260927 21:45:13.110141 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.087364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.0.202.188:46013)
0927 21:45:13.088202 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.088220 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.088276 (+ 56us) server_negotiation.cc:407] Connection header received
0927 21:45:13.090131 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.090155 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.090617 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.091078 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.092773 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.094153 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.095801 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.096735 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.096989 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.101343 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.101352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.101368 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.101415 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.104907 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.105860 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.105866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.105873 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.106057 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.106550 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.106554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.106557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.107423 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.107799 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.108341 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.108783 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":355,"threads_started":1}
I20260927 21:45:13.114704 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.096958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.190:33563 (local address 127.0.0.1:50206)
0927 21:45:13.099160 (+ 2202us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:13.099202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:13.099414 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:13.099838 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:13.099845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:13.100621 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:13.100874 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.100883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.102268 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.102275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.103419 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.103431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.104499 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.104506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.104722 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.105336 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:13.105361 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:13.108042 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:13.111479 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.111490 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.111499 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.111997 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.112480 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.112485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.112489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.112695 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.113360 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:13.113369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:13.113596 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 21:45:13.113814 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 21:45:13.114214 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2011,"thread_start_us":687,"threads_started":1}
I20260927 21:45:13.114722 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.096912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.0.202.190:33563)
0927 21:45:13.098886 (+ 1974us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.098891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.099312 (+ 421us) server_negotiation.cc:407] Connection header received
0927 21:45:13.099569 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.099574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.099655 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.099773 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.101034 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.102115 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.103803 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.104245 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.104449 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.108251 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.108256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.108259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.108335 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.111300 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.112135 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.112141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.112144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.112339 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.112833 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.112838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.112840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.113198 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.113315 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.114172 (+ 857us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.114342 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1870,"thread_start_us":1717,"threads_started":1}
I20260927 21:45:13.116516 23853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.190@KRBTEST.COM
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:13.130900 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.117305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.189:46629 (local address 127.0.0.1:35216)
0927 21:45:13.117541 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:13.117556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:13.117638 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:13.117970 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:13.117975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:13.118386 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:13.118516 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.118524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.119750 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.119754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.120697 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.120709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.121777 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.121781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.121924 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.122560 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:13.122583 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:13.125190 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:13.128367 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.128375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.128379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.128770 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.129179 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.129184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.129188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.129339 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.129881 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:13.129886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:13.130066 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 21:45:13.130235 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 21:45:13.130548 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 21:45:13.129941 23704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 4
I20260927 21:45:13.131651 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.113614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.202.190:33563)
0927 21:45:13.114303 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.114311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.114337 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:45:13.114396 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.114402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.114524 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.114658 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.116206 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.117230 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.118973 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.119400 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.119495 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.124987 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.124994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.124998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.125035 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.128204 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.129040 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.129047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.129051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.129213 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.129614 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.129620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.129623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.130015 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.130303 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.130672 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.130983 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":279,"threads_started":1}
I20260927 21:45:13.131945 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.117311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.0.202.189:46629)
0927 21:45:13.117520 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.117526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.117629 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:45:13.117722 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.117728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.117811 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.117929 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.118678 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.119604 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.120855 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.121616 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.121775 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.125422 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.125428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.125431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.125468 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.128189 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.128899 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.128906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.128909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.129047 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.129466 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.129470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.129472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.129755 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.129921 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.131220 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.131477 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:45:13.133244 23853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.190@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.188@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.189@KRBTEST.COM
I20260927 21:45:13.150821 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.132101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.0.202.188:46013)
0927 21:45:13.132404 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.132413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.132862 (+ 449us) server_negotiation.cc:407] Connection header received
0927 21:45:13.133205 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.133212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.133304 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.133411 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.134554 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.135767 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.137681 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.138264 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.138410 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.143645 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.143650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.143654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.143686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.147586 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.148535 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.148544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.148548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.148713 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.149187 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.149191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.149194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.149581 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.149736 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.150230 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.150469 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 21:45:13.150847 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.132209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59950)
0927 21:45:13.132728 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:13.132770 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:13.132884 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:13.133591 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:13.133596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:13.134169 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:13.134394 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.134405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.135911 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.135919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.137490 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.137506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.138427 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.138434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.138659 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.139400 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:13.139427 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:13.143396 (+ 3969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:13.147784 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.147790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.147793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.148363 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.148869 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.148873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.148877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.149018 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.149818 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:13.149824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:13.149979 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:45:13.150125 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 21:45:13.150432 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418}
I20260927 21:45:13.151870 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.134159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59962)
0927 21:45:13.134363 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:13.134386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:13.134503 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:13.135369 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:13.135380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:13.135943 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:13.136131 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.136142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.137856 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.137862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.139074 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.139086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.139766 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.139771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.139994 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.140621 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:13.140648 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:13.144509 (+ 3861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:13.148889 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.148896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.148901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.149447 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.149876 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.149881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.149884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.150033 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.150785 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:13.150789 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:13.150938 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 21:45:13.151090 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 21:45:13.151444 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:45:13.152823 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.134036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.0.202.188:46013)
0927 21:45:13.135056 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.135063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.135087 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:45:13.135131 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.135136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.135204 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.135313 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.136363 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.137713 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.139207 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.139608 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.139746 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.144743 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.144751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.144756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.144795 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.148754 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.149575 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.149584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.149588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.149730 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.150163 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.150167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.150170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.150590 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.150736 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.152243 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.152509 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":630,"threads_started":1}
I20260927 21:45:13.153585 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.133546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.0.202.189:46629)
0927 21:45:13.133715 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.133720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.134640 (+ 920us) server_negotiation.cc:407] Connection header received
0927 21:45:13.134709 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.134717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.134787 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.134875 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.136461 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.137178 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.138776 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.139575 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.139707 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.145420 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.145428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.145432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.145488 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.148982 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.149834 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.149841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.149848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.150028 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.150523 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.150530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.150535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.150919 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.151222 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.152961 (+ 1739us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.153203 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:45:13.157980 23853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: Bootstrap starting.
I20260927 21:45:13.158066 23704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067: Bootstrap starting.
I20260927 21:45:13.158981 810 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:45:13.161590 23704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:13.161741 23853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:13.162911 23704 log.cc:824] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:13.163028 23853 log.cc:824] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:13.166785 23853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: No bootstrap required, opened a new log
I20260927 21:45:13.162174 23679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "135fcf8d390a427cbe53606bcb457067"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50218
I20260927 21:45:13.166851 23704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067: No bootstrap required, opened a new log
I20260927 21:45:13.174583 23704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.174772 23853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.175052 23704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135fcf8d390a427cbe53606bcb457067, State: Initialized, Role: FOLLOWER
I20260927 21:45:13.175217 23853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0875acdf678b4371a04966c52b14be55, State: Initialized, Role: FOLLOWER
I20260927 21:45:13.175599 23704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.175654 23853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.176671 23863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.177219 23863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:13.176838 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.177497 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:13.178057 23853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:13.178086 23704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:13.178190 23679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:45:13.178369 23679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:13.182679 23679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.185873 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "135fcf8d390a427cbe53606bcb457067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0875acdf678b4371a04966c52b14be55"
I20260927 21:45:13.186509 23828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:13.191272 23828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135fcf8d390a427cbe53606bcb457067 in term 1.
I20260927 21:45:13.192155 23656 leader_election.cc:302] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0875acdf678b4371a04966c52b14be55, 135fcf8d390a427cbe53606bcb457067; no voters:
I20260927 21:45:13.193040 23679 leader_election.cc:288] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [CANDIDATE]: Term 1 election: Requested vote from peers 039e06be2332495684361766e8b5baea (127.0.202.189:46629), 0875acdf678b4371a04966c52b14be55 (127.0.202.188:46013)
I20260927 21:45:13.193080 23864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:45:13.203930 23864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [term 1 LEADER]: Becoming Leader. State: Replica: 135fcf8d390a427cbe53606bcb457067, State: Running, Role: LEADER
I20260927 21:45:13.204775 23864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.200446 23786 master_runner.cc:186] Error getting master registration for 127.0.202.190:33563: OK, instance_id { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" instance_seqno: 1790545511983762 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:45:13.218354 23786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:45:13.219215 23876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:45:13.219442 23876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:45:13.220284 23879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135fcf8d390a427cbe53606bcb457067. Latest consensus state: current_term: 1 leader_uuid: "135fcf8d390a427cbe53606bcb457067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.220889 23879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: This master's current role is: LEADER
W20260927 21:45:13.221853 23658 tablet.cc:2369] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:45:13.223469 23892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:45:13.239352 23778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } attempt: 5
I20260927 21:45:13.242888 23892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:45:13 dist-test-slave-25dq krb5kdc[23624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.189@KRBTEST.COM for kudu/127.0.202.188@KRBTEST.COM
I20260927 21:45:13.259850 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.240127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.202.188:46013 (local address 127.0.0.1:59984)
0927 21:45:13.240664 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 21:45:13.240687 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:45:13.240809 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:45:13.241733 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:45:13.241739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:45:13.242339 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:45:13.242567 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.242581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.244332 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.244338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.245906 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:45:13.245925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.247357 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.247364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:45:13.247684 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.248420 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:45:13.248446 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:45:13.251803 (+ 3357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:45:13.256499 (+ 4696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.256505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.256512 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.257155 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.257666 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:45:13.257671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:45:13.257676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:45:13.257866 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:45:13.258707 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:45:13.258712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:45:13.258890 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 21:45:13.259052 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 21:45:13.259381 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373}
I20260927 21:45:13.261089 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.241037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.0.202.188:46013)
0927 21:45:13.241244 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.241251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.241271 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:45:13.241334 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.241340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.241422 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.241539 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.242757 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.244021 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.246156 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.246980 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.247111 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.252073 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.252081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.252086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.252132 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.256275 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.257317 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.257323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.257327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.257517 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.258014 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.258019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.258023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.258478 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.258639 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.260407 (+ 1768us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.260678 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:45:13.270838 23778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea: Bootstrap starting.
I20260927 21:45:13.275010 23778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea: Neither blocks nor log segments found. Creating new log.
I20260927 21:45:13.279938 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.230903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.202.189:46629)
0927 21:45:13.231075 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.231081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.245233 (+ 14152us) server_negotiation.cc:407] Connection header received
0927 21:45:13.245733 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.245742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.245846 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.245925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.248533 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.249847 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.252614 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.253254 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.253361 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.260827 (+ 7466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.260834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.260839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.260884 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.268623 (+ 7739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.277297 (+ 8674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.277304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.277308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.277511 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.277951 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.277959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.277962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.278353 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.278656 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.279243 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.279534 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:45:13.281919 23778 log.cc:824] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea: Log is configured to *not* fsync() on all Append() calls
I20260927 21:45:13.270902 23637 master_runner.cc:186] Error getting master registration for 127.0.202.189:46629: OK, instance_id { permanent_uuid: "039e06be2332495684361766e8b5baea" instance_seqno: 1790545512507851 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:45:13.290107 23637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:45:13.296536 23828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [term 1 FOLLOWER]: Refusing update from remote peer 135fcf8d390a427cbe53606bcb457067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:45:13.297197 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.229535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.0.202.190:33563)
0927 21:45:13.229771 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.229778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.230442 (+ 664us) server_negotiation.cc:407] Connection header received
0927 21:45:13.231005 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.231011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.231096 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.231196 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.239090 (+ 7894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.240214 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.252241 (+ 12027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.252928 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.253030 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.284721 (+ 31691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.284731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.284738 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.284791 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.290256 (+ 5465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.291468 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.291502 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.291505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.291719 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.292480 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.292485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.292488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.293069 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.293452 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.294594 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.296640 (+ 2046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:45:13.297609 23879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:45:13.299405 23778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea: No bootstrap required, opened a new log
W20260927 21:45:13.311537 23807 tablet.cc:2369] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:45:13.318693 23778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.319218 23778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039e06be2332495684361766e8b5baea, State: Initialized, Role: FOLLOWER
I20260927 21:45:13.319855 23778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } }
I20260927 21:45:13.322417 23863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135fcf8d390a427cbe53606bcb457067. Latest consensus state: current_term: 1 leader_uuid: "135fcf8d390a427cbe53606bcb457067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.322742 23863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:13.324743 23899 mvcc.cc:205] Tried to move back new op lower bound from 7334074422447022080 to 7334074422094290944. Current Snapshot: MvccSnapshot[applied={T|T < 7334074422447022080}]
I20260927 21:45:13.327903 23879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "135fcf8d390a427cbe53606bcb457067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.328179 23879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:13.329082 23863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "135fcf8d390a427cbe53606bcb457067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.329420 23863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:13.329669 23901 mvcc.cc:205] Tried to move back new op lower bound from 7334074422447022080 to 7334074422094290944. Current Snapshot: MvccSnapshot[applied={T|T < 7334074422447022080}]
I20260927 21:45:13.331779 23892 catalog_manager.cc:1347] Generated new cluster ID: af056ffa21b14ddd816ed6afdf0ae65f
I20260927 21:45:13.331884 23892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:45:13.333127 23903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.333585 23903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:13.335557 23778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:45:13.335975 23752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:45:13.352420 23753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "135fcf8d390a427cbe53606bcb457067" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "039e06be2332495684361766e8b5baea"
I20260927 21:45:13.355520 23752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [term 1 FOLLOWER]: Refusing update from remote peer 135fcf8d390a427cbe53606bcb457067: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:45:13.358098 23753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135fcf8d390a427cbe53606bcb457067 in term 1.
I20260927 21:45:13.360548 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:13.237901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.0.202.188:46013)
0927 21:45:13.238110 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.238116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.281674 (+ 43558us) server_negotiation.cc:407] Connection header received
0927 21:45:13.282731 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.282740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.282875 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.282974 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.288420 (+ 5446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.289574 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.309843 (+ 20269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:13.310852 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:13.311026 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:13.320218 (+ 9192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:45:13.320226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:45:13.320230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:45:13.320279 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:45:13.331066 (+ 10787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.353193 (+ 22127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.353202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.353207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.353421 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:45:13.354261 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:45:13.354267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:45:13.354270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:45:13.354695 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:45:13.354965 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:13.359825 (+ 4860us) server_negotiation.cc:299] Negotiation successful
0927 21:45:13.360109 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:45:13.363626 23864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:45:13.432049 23903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135fcf8d390a427cbe53606bcb457067. Latest consensus state: current_term: 1 leader_uuid: "135fcf8d390a427cbe53606bcb457067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.432591 23903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:13.438825 23912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "135fcf8d390a427cbe53606bcb457067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.439414 23912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:45:13.461638 23920 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:45:13.474627 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "135fcf8d390a427cbe53606bcb457067" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fcf8d390a427cbe53606bcb457067" member_type: VOTER last_known_addr { host: "127.0.202.190" port: 33563 } } peers { permanent_uuid: "039e06be2332495684361766e8b5baea" member_type: VOTER last_known_addr { host: "127.0.202.189" port: 46629 } } peers { permanent_uuid: "0875acdf678b4371a04966c52b14be55" member_type: VOTER last_known_addr { host: "127.0.202.188" port: 46013 } } }
I20260927 21:45:13.474987 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067 [sys.catalog]: This master's current role is: LEADER
I20260927 21:45:13.480425 23892 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:45:13.482640 23892 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:45:13.505220 23892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135fcf8d390a427cbe53606bcb457067: Generated new TSK 0
I20260927 21:45:13.506481 23892 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:45:13.611950 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.596000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.0.202.189:46629)
0927 21:45:13.596283 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.596312 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.609155 (+ 12843us) server_negotiation.cc:407] Connection header received
0927 21:45:13.610137 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.610144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.610235 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.610317 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.611682 (+ 1365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35240: BlockingRecv error: recv got EOF from 127.0.0.1:35240 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 21:45:13.614085 23859 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.595777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.0.202.190:33563)
0927 21:45:13.596041 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.596052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.609327 (+ 13275us) server_negotiation.cc:407] Connection header received
0927 21:45:13.609615 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.609623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.609765 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.609876 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.613754 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50230: BlockingRecv error: recv got EOF from 127.0.0.1:50230 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 21:45:13.614785 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.602068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.0.202.188:46013)
0927 21:45:13.602287 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.602296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.612904 (+ 10608us) server_negotiation.cc:407] Connection header received
0927 21:45:13.613186 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.613194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.613296 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.613381 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.614422 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59998: BlockingRecv error: recv got EOF from 127.0.0.1:59998 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 21:45:13.623140 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.620789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.0.202.189:46629)
0927 21:45:13.620989 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.620995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.621465 (+ 470us) server_negotiation.cc:407] Connection header received
0927 21:45:13.621712 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.621719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.621869 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.621995 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.622869 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35242: BlockingRecv error: recv got EOF from 127.0.0.1:35242 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 21:45:13.633152 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.630801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.0.202.189:46629)
0927 21:45:13.630998 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.631005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.631550 (+ 545us) server_negotiation.cc:407] Connection header received
0927 21:45:13.631767 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.631775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.631879 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.631997 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.632918 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35244: BlockingRecv error: recv got EOF from 127.0.0.1:35244 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 21:45:13.640699 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.638450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.0.202.189:46629)
0927 21:45:13.638629 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.638635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.639135 (+ 500us) server_negotiation.cc:407] Connection header received
0927 21:45:13.639306 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.639313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.639423 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.639542 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.640398 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35260: BlockingRecv error: recv got EOF from 127.0.0.1:35260 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 21:45:13.650470 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.648098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.0.202.189:46629)
0927 21:45:13.648353 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.648361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.648886 (+ 525us) server_negotiation.cc:407] Connection header received
0927 21:45:13.649089 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.649097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.649214 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.649337 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.650206 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35276: BlockingRecv error: recv got EOF from 127.0.0.1:35276 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260927 21:45:13.668557 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.666307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.202.189:46629)
0927 21:45:13.666497 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.666504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.666993 (+ 489us) server_negotiation.cc:407] Connection header received
0927 21:45:13.667172 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.667179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.667286 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.667410 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.668256 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35278: BlockingRecv error: recv got EOF from 127.0.0.1:35278 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 21:45:13.690672 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.688310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.0.202.189:46629)
0927 21:45:13.688515 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.688522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.689041 (+ 519us) server_negotiation.cc:407] Connection header received
0927 21:45:13.689266 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.689272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.689393 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.689534 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.690376 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35280: BlockingRecv error: recv got EOF from 127.0.0.1:35280 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 21:45:13.732776 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.730329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.202.189:46629)
0927 21:45:13.730543 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.730549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.731094 (+ 545us) server_negotiation.cc:407] Connection header received
0927 21:45:13.731303 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.731311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.731418 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.731547 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.732472 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35290: BlockingRecv error: recv got EOF from 127.0.0.1:35290 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 21:45:13.802717 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.800479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.0.202.189:46629)
0927 21:45:13.800705 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.800712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.801038 (+ 326us) server_negotiation.cc:407] Connection header received
0927 21:45:13.801307 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.801312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.801397 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.801497 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.802465 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35302: BlockingRecv error: recv got EOF from 127.0.0.1:35302 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 21:45:13.938802 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:13.937094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.0.202.189:46629)
0927 21:45:13.937292 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:13.937301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:13.937326 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:45:13.937576 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:13.937582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:13.937670 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:13.937784 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:13.938525 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35316: BlockingRecv error: recv got EOF from 127.0.0.1:35316 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 21:45:14.206846 23783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:45:14.204865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.0.202.189:46629)
0927 21:45:14.205115 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:14.205122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:14.205315 (+ 193us) server_negotiation.cc:407] Connection header received
0927 21:45:14.205548 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:14.205553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:14.205630 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:14.205727 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:45:14.206608 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:45:14.221760 23876 catalog_manager.cc:1259] Loaded cluster ID: af056ffa21b14ddd816ed6afdf0ae65f
I20260927 21:45:14.221854 23876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: loading cluster ID for follower catalog manager: success
I20260927 21:45:14.224397 23876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: acquiring CA information for follower catalog manager: success
I20260927 21:45:14.226193 23876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0875acdf678b4371a04966c52b14be55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:45:14.466470 23920 catalog_manager.cc:1259] Loaded cluster ID: af056ffa21b14ddd816ed6afdf0ae65f
I20260927 21:45:14.466593 23920 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea: loading cluster ID for follower catalog manager: success
I20260927 21:45:14.469300 23920 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea: acquiring CA information for follower catalog manager: success
I20260927 21:45:14.471194 23920 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 039e06be2332495684361766e8b5baea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:45:14.733867 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:14.727360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.0.202.189:46629)
0927 21:45:14.728192 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:14.728199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:14.728215 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:45:14.728276 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:14.728280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:14.728490 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:14.728628 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:14.729067 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.730007 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.731389 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.731855 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.732000 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:14.732584 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:14.733189 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:14.733379 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:14.733431 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:45:14.733520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":487,"threads_started":1}
I20260927 21:45:14.746685 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:14.740571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.0.202.188:46013)
0927 21:45:14.741511 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:14.741516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:14.741530 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:45:14.741619 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:14.741623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:14.741773 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:14.741951 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:14.742352 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.743098 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.744494 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.744917 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.745067 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:14.745557 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:14.746072 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:14.746240 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:14.746308 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:45:14.746394 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":716,"threads_started":1}
I20260927 21:45:14.762724 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:14.753751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.0.202.190:33563)
0927 21:45:14.754186 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:14.754194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:14.754217 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:45:14.754354 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:14.754361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:14.754604 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:14.755350 (+ 746us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:14.755547 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.756710 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.758908 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.759561 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.760224 (+ 663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:14.760434 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:14.760890 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:14.761758 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:14.761811 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:45:14.762384 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":327,"threads_started":1}
I20260927 21:45:14.763412 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:14.753429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.0.202.189:46629)
0927 21:45:14.753601 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:14.753607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:14.754662 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 21:45:14.754950 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:14.754959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:14.755058 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:14.755126 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:14.755840 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.757713 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.760956 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.761613 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.761723 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:14.762590 (+ 867us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:14.762725 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:14.762788 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:14.763038 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 21:45:14.763115 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:45:14.768193 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:45:14.753786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.0.202.188:46013)
0927 21:45:14.753954 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:45:14.753960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:45:14.761552 (+ 7592us) server_negotiation.cc:407] Connection header received
0927 21:45:14.761717 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:45:14.761724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:45:14.761817 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:45:14.761946 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:45:14.762346 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.763447 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.766336 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:45:14.766914 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:45:14.767072 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:45:14.767481 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:45:14.767592 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:45:14.767695 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:45:14.767838 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 21:45:14.767919 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:45:14.774304 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23637
I20260927 21:45:14.775530 23699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:45:14.892884 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23637
I20260927 21:45:14.914207 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23711
I20260927 21:45:14.915408 23773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:45:15.025098 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23711
I20260927 21:45:15.045408 810 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Z3URy/build/asan/bin/kudu pid 23786
I20260927 21:45:15.046631 23848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:45:15.156100 810 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Z3URy/build/asan/bin/kudu with pid 23786
2026-09-27T21:45:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5010 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10269 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (379316 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST